Max level shown:
12025-04-30T06:45:53.052ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_crdb
22025-04-30T06:45:53.053ZDEBGtest_multiple_snapshots: Starting CRDB
32025-04-30T06:45:53.085ZINFOtest_multiple_snapshots: cockroach temporary directory: /var/tmp/omicron_tmp/.tmp254w9v
42025-04-30T06:45:53.086ZINFOtest_multiple_snapshots: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmp254w9v/data)
52025-04-30T06:45:53.114ZINFOtest_multiple_snapshots: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmp254w9v/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmp254w9v/listen-url
62025-04-30T06:45:53.115ZINFOtest_multiple_snapshots: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=7 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:45:54.600ZINFOtest_multiple_snapshots: cockroach pid: 27786
82025-04-30T06:45:54.600ZINFOtest_multiple_snapshots: cockroach listen URL: postgresql://root@[::1]:55912/omicron?sslmode=disable
92025-04-30T06:45:54.601ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_clickhouse
102025-04-30T06:45:54.601ZDEBGtest_multiple_snapshots: Starting Clickhouse
112025-04-30T06:45:55.318ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:45:55.318ZDEBGtest_multiple_snapshots: Starting Management Gateway
132025-04-30T06:45:55.442ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:45:55.442ZDEBGtest_multiple_snapshots: Starting Dendrite for switch0
152025-04-30T06:45:55.489ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:45:55.489ZDEBGtest_multiple_snapshots: Starting mgd for switch0
172025-04-30T06:45:55.646ZDEBGtest_multiple_snapshots: mgd port is 59043
182025-04-30T06:45:55.646ZDEBGtest_multiple_snapshots (init_with_steps): Running step record_switch_dns
192025-04-30T06:45:55.646ZDEBGtest_multiple_snapshots: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:45:55.646ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_internal_dns
212025-04-30T06:45:55.647ZINFOtest_multiple_snapshots (store): opening sled database path = /var/tmp/omicron_tmp/.tmppdKYdB
222025-04-30T06:45:55.777ZINFOtest_multiple_snapshots (store): pruning trees for generations newer than 0
232025-04-30T06:45:55.777ZINFOtest_multiple_snapshots (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:45:55.777ZINFOtest_multiple_snapshots (dns): DNS server bound to address kind = dns local_address = [::1]:58772
252025-04-30T06:45:55.778ZDEBGtest_multiple_snapshots (http): registered endpoint kind = dns local_addr = [::1]:35840 method = GET path = /config versions = all
262025-04-30T06:45:55.778ZDEBGtest_multiple_snapshots (http): registered endpoint kind = dns local_addr = [::1]:35840 method = PUT path = /config versions = all
272025-04-30T06:45:55.779ZINFOtest_multiple_snapshots (http): listening kind = dns local_addr = [::1]:35840
282025-04-30T06:45:55.780ZDEBGtest_multiple_snapshots (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:35840
292025-04-30T06:45:55.781ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_external_dns
302025-04-30T06:45:55.781ZINFOtest_multiple_snapshots (store): opening sled database path = /var/tmp/omicron_tmp/.tmppT2Iz7
312025-04-30T06:45:55.793ZINFOtest_multiple_snapshots (store): pruning trees for generations newer than 0
322025-04-30T06:45:55.793ZINFOtest_multiple_snapshots (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:45:55.793ZINFOtest_multiple_snapshots (dns): DNS server bound to address kind = dns local_address = [::1]:33090
342025-04-30T06:45:55.793ZDEBGtest_multiple_snapshots (http): registered endpoint kind = dns local_addr = [::1]:42239 method = GET path = /config versions = all
352025-04-30T06:45:55.793ZDEBGtest_multiple_snapshots (http): registered endpoint kind = dns local_addr = [::1]:42239 method = PUT path = /config versions = all
362025-04-30T06:45:55.793ZINFOtest_multiple_snapshots (http): listening kind = dns local_addr = [::1]:42239
372025-04-30T06:45:55.794ZDEBGtest_multiple_snapshots (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:42239
382025-04-30T06:45:55.794ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_nexus_internal
392025-04-30T06:45:55.794ZDEBGtest_multiple_snapshots: Starting Nexus (internal API)
402025-04-30T06:45:55.795ZINFOtest_multiple_snapshots: setting up nexus server
412025-04-30T06:45:55.816ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Action
422025-04-30T06:45:55.816ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:45:55.816ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:45:55.816ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Database
462025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Fleet
482025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Inventory
492025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:45:55.817ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Project
582025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Disk
592025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Instance
642025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = IpPool
652025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Vpc
672025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:45:55.818ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Image
752025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Certificate
802025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Rack
852025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SshKey
872025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Silo
882025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Sled
942025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = Service
1012025-04-30T06:45:55.819ZINFOtest_multiple_snapshots (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:45:55.820ZINFOtest_multiple_snapshots (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:45:55.894ZINFOtest_multiple_snapshots (ServerContext): Setting up resolver using DNS address: [::1]:58772
1042025-04-30T06:45:55.894ZINFOtest_multiple_snapshots (DnsResolver): new DNS resolver addresses = [[::1]:58772]
1052025-04-30T06:45:55.895ZINFOtest_multiple_snapshots (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:55912/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 55912,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:45:55.899ZDEBGtest_multiple_snapshots (ServerContext): registered USDT probes
1072025-04-30T06:45:56.008ZINFOtest_multiple_snapshots (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:45:56.111ZDEBGtest_multiple_snapshots (clickhouse-client): registered USDT probes id = 00d7704a-1cdb-4897-8b51-2c76fd0945fd
1092025-04-30T06:45:56.237ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /bgtasks versions = all
1102025-04-30T06:45:56.237ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:45:56.237ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:45:56.237ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /demo-saga versions = all
1202025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:45:56.238ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:45:56.239ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:45:56.240ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:45:56.240ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /metrics/producers versions = all
1442025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /mgs-updates versions = all
1452025-04-30T06:45:56.241ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /sagas versions = all
1522025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:45:56.242ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /sleds/add versions = all
1572025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = GET path = /v1/ping versions = all
1612025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:45:56.243ZDEBGtest_multiple_snapshots (dropshot_internal): registered endpoint local_addr = 127.0.0.1:55015 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:45:56.244ZINFOtest_multiple_snapshots (dropshot_internal): listening local_addr = 127.0.0.1:55015
1642025-04-30T06:45:56.244ZDEBGtest_multiple_snapshots (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:55015
1652025-04-30T06:45:56.244ZINFOtest_multiple_snapshots (SEC): SEC running sec_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
1662025-04-30T06:45:56.245ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:45:56.245ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:45:56.250ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:45:56.251ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in users
1702025-04-30T06:45:56.254ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:45:56.254ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:45:56.254ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:45:56.255ZINFOtest_multiple_snapshots (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:45:56.258ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 4f 28 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 O(..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:54085 req_id = 79b2d8ef-b8ed-4fb4-88dd-d9b1e5780c1c
1752025-04-30T06:45:56.259ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20264,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54085 req_id = 79b2d8ef-b8ed-4fb4-88dd-d9b1e5780c1c
1762025-04-30T06:45:56.274ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20264, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54085 req_id = 79b2d8ef-b8ed-4fb4-88dd-d9b1e5780c1c
1772025-04-30T06:45:56.344ZINFOtest_multiple_snapshots (DataLoader): created 5 built-in users
1782025-04-30T06:45:56.347ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:45:56.353ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:45:56.365ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:45:56.365ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in roles
1822025-04-30T06:45:56.365ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:45:56.365ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:45:56.366ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:45:56.395ZINFOtest_multiple_snapshots (DataLoader): created 10 built-in roles
1862025-04-30T06:45:56.395ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:45:56.395ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:45:56.398ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:45:56.398ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:45:56.398ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:45:56.398ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:45:56.398ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:45:56.708ZINFOtest_multiple_snapshots (DataLoader): created 4 built-in role assignments
1942025-04-30T06:45:56.708ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:45:56.708ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:45:56.710ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:45:56.710ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in silos
1982025-04-30T06:45:56.710ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:45:56.710ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:45:56.710ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:45:56.796ZINFOtest_multiple_snapshots (DataLoader): created 2 built-in silos
2022025-04-30T06:45:56.796ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:45:56.796ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:45:56.797ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:45:56.810ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:45:56.810ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:45:56.813ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:45:56.813ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in projects
2092025-04-30T06:45:56.813ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:45:56.813ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:45:56.813ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:45:56.822ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:45:56.822ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:45:56.822ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:45:56.822ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:45:56.823ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:45:56.894ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:45:56.894ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:45:56.894ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:45:56.895ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:45:56.901ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:45:56.905ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:45:56.905ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:45:56.905ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:45:56.905ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:45:56.905ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:45:56.905ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:45:56.908ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:45:56.908ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:45:56.908ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:45:56.908ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:45:56.911ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:45:56.915ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:45:56.915ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:45:56.915ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:45:56.915ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:45:56.933ZINFOtest_multiple_snapshots (DataLoader): created built-in services project
2382025-04-30T06:45:56.933ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:45:56.933ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:45:56.937ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:45:56.937ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:45:56.937ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:45:56.937ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:45:56.938ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:45:56.941ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:45:56.941ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:45:56.941ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:45:56.944ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:45:56.944ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:45:56.944ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:45:56.944ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:45:56.945ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:45:56.946ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:45:56.947ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:45:56.947ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:45:56.948ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:45:56.950ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:45:56.950ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:45:56.950ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:45:56.950ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:45:56.952ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:45:56.960ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:45:56.960ZTRACtest_multiple_snapshots (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:45:56.961ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:45:56.961ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:45:56.961ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:45:56.961ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:45:56.961ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:45:56.964ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:45:56.964ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:45:56.964ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:45:56.966ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:45:56.968ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:45:56.968ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:45:56.968ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:45:56.969ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:45:56.972ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:45:56.980ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:45:56.980ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:45:56.980ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:45:56.981ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:45:56.997ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:45:56.997ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:45:56.997ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:45:56.997ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:45:56.998ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:45:57.001ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:45:57.001ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:45:57.001ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:45:57.001ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:45:57.004ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:45:57.004ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:45:57.004ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:45:57.005ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:45:57.008ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:45:57.016ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:45:57.016ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:45:57.016ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:45:57.016ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:45:57.024ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:45:57.024ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:45:57.025ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:45:57.032ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:45:57.032ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:45:57.034ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:45:57.039ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:45:57.039ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:45:57.039ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:45:57.039ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:45:57.040ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:45:57.043ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:45:57.043ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:45:57.043ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:45:57.044ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:45:57.048ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:45:57.048ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:45:57.048ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:45:57.053ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:45:57.069ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:45:57.078ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:45:57.078ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:45:57.078ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:45:57.079ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:45:57.157ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:45:57.157ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:45:57.157ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:45:57.157ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:45:57.162ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:45:57.176ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:45:57.176ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:45:57.176ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:45:57.178ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:45:57.184ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:45:57.184ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:45:57.184ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:45:57.185ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:45:57.193ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:45:57.202ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:45:57.202ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:45:57.202ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:45:57.203ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:45:57.267ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:45:57.267ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:45:57.267ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:45:57.267ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:45:57.267ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:45:57.272ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:45:57.272ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:45:57.272ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:45:57.273ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:45:57.277ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:45:57.277ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:45:57.277ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:45:57.277ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:45:57.280ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:45:57.287ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:45:57.287ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:45:57.287ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:45:57.287ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:45:57.362ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:45:57.363ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:45:57.363ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:45:57.363ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:45:57.376ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:45:57.376ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:45:57.377ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:45:57.382ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:45:57.382ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:45:57.383ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:45:57.388ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:45:57.388ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:45:57.389ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:45:57.389ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:45:57.389ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:45:57.394ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:45:57.394ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:45:57.394ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:45:57.394ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:45:57.399ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:45:57.399ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:45:57.399ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:45:57.400ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:45:57.405ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:45:57.410ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:45:57.410ZTRACtest_multiple_snapshots (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:45:57.410ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:45:57.410ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:45:57.410ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:45:57.410ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:45:57.411ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:45:57.420ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:45:57.420ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:45:57.420ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:45:57.420ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:45:57.420ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:45:57.420ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:45:57.420ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:45:57.421ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:45:57.431ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:45:57.440ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:45:57.440ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:45:57.440ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:45:57.440ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:45:57.498ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:45:57.498ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:45:57.498ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:45:57.498ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:45:57.500ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:45:57.503ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:45:57.503ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:45:57.503ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:45:57.504ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:45:57.510ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:45:57.510ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:45:57.510ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:45:57.511ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:45:57.515ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:45:57.521ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:45:57.521ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:45:57.521ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:45:57.522ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:45:57.644ZINFOtest_multiple_snapshots (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:45:57.644ZDEBGtest_multiple_snapshots (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:45:57.644ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:45:57.644ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:45:57.645ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:45:57.656ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:45:57.656ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:45:57.657ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:45:57.659ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:45:57.659ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:45:57.660ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:45:57.662ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:45:57.662ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:45:57.662ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:45:57.662ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:45:57.662ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:45:57.667ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:45:57.667ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:45:57.667ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:45:57.668ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:45:57.670ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:45:57.670ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:45:57.670ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:45:57.670ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:45:57.674ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:45:57.679ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:45:57.679ZTRACtest_multiple_snapshots (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:45:57.679ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:45:57.679ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:45:57.680ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:45:57.687ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:45:57.687ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:45:57.687ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:45:57.687ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:45:57.687ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:45:57.690ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:45:57.690ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:45:57.690ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:45:57.690ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:45:57.692ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:45:57.692ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:45:57.692ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:45:57.693ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:45:57.695ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:45:57.700ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:45:57.708ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:45:57.709ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:45:57.709ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:45:57.709ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:45:57.709ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:45:57.711ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:45:57.711ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:45:57.711ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:45:57.712ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:45:57.715ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:45:57.715ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:45:57.715ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:45:57.716ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:45:57.718ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:45:57.724ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:45:57.724ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:45:57.724ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:45:57.724ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:45:57.734ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:45:57.734ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:45:57.734ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:45:57.737ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:45:57.737ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:45:57.737ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:45:57.737ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:45:57.738ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:45:57.741ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:45:57.741ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:45:57.741ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:45:57.742ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:45:57.744ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:45:57.744ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:45:57.744ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:45:57.745ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:45:57.747ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:45:57.752ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:45:57.758ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:45:57.758ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:45:57.758ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:45:57.758ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:45:57.759ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:45:57.762ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:45:57.762ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:45:57.762ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:45:57.763ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:45:57.765ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:45:57.765ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:45:57.765ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:45:57.766ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:45:57.768ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:45:57.774ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:45:57.774ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:45:57.774ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:45:57.775ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:45:57.783ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:45:57.783ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:45:57.784ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:45:57.787ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:45:57.787ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:45:57.787ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:45:57.787ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:45:57.788ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:45:57.790ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:45:57.790ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:45:57.790ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:45:57.792ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:45:57.794ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:45:57.794ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:45:57.794ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:45:57.794ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:45:57.812ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:45:57.822ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:45:57.827ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:45:57.827ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:45:57.827ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:45:57.827ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:45:57.828ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:45:57.830ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:45:57.830ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:45:57.830ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:45:57.834ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:45:57.836ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:45:57.836ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:45:57.836ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:45:57.837ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:45:57.839ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:45:57.847ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:45:57.847ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:45:57.848ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:45:57.848ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:45:57.857ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:45:57.858ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:45:57.858ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:45:57.862ZINFOtest_multiple_snapshots (DataLoader): created built-in services vpc subnets
5612025-04-30T06:45:57.862ZINFOtest_multiple_snapshots (DataLoader): created built-in services vpc
5622025-04-30T06:45:57.862ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:45:57.862ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:45:57.864ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:45:57.864ZDEBGtest_multiple_snapshots (DataLoader): attempting to create silo users
5662025-04-30T06:45:57.864ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:45:57.864ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:45:57.865ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:45:57.874ZINFOtest_multiple_snapshots (DataLoader): created 2 silo users
5702025-04-30T06:45:57.874ZDEBGtest_multiple_snapshots (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:45:57.874ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:45:57.874ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:45:57.875ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:45:57.880ZINFOtest_multiple_snapshots (DataLoader): created 2 silo user role assignments
5752025-04-30T06:45:57.880ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:45:57.880ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:45:57.882ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:45:57.887ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:45:57.887ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:45:57.888ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:45:57.907ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:45:57.907ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:45:57.907ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:45:57.907ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:45:57.908ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:45:57.911ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:45:57.916ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:45:57.916ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:45:57.916ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:45:57.917ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:45:57.927ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:45:57.927ZTRACtest_multiple_snapshots (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:45:57.927ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:45:57.927ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:45:57.928ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:45:57.933ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:45:57.937ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:45:57.937ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:45:57.937ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:45:57.938ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:45:58.006ZTRACtest_multiple_snapshots (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:45:58.006ZDEBGtest_multiple_snapshots (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:45:58.007ZDEBGtest_multiple_snapshots (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:45:58.011ZINFOtest_multiple_snapshots (nexus): populate complete
6052025-04-30T06:45:58.011ZINFOtest_multiple_snapshots (nexus): activating background tasks
6062025-04-30T06:45:58.015ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_sled1
6072025-04-30T06:45:58.015ZINFOtest_multiple_snapshots (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:45:58.090ZINFOtest_multiple_snapshots (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:45:58.090ZINFOtest_multiple_snapshots (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:45:58.091ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:45:58.091ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:45:58.091ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:45:58.091ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:45:58.092ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:45:58.093ZINFOtest_multiple_snapshots (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpcGE6OU/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6162025-04-30T06:45:58.093ZDEBGtest_multiple_snapshots (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpcGE6OU/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6172025-04-30T06:45:58.093ZINFOtest_multiple_snapshots (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpLSfmRQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6182025-04-30T06:45:58.093ZDEBGtest_multiple_snapshots (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpLSfmRQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6192025-04-30T06:45:58.158ZDEBGtest_multiple_snapshots (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:45115 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6202025-04-30T06:45:58.158ZINFOtest_multiple_snapshots (dropshot (Repo Depot)): listening local_addr = [::1]:45115 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6212025-04-30T06:45:58.159ZDEBGtest_multiple_snapshots (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:45115 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6222025-04-30T06:45:58.165ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6232025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6242025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6252025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6262025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6272025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6282025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6292025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6302025-04-30T06:45:58.166ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6312025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6322025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6332025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6342025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6352025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6362025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6372025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6382025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6392025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6402025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6412025-04-30T06:45:58.167ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6422025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6432025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6442025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6452025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6462025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6472025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6482025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6492025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6502025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6512025-04-30T06:45:58.168ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6522025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6532025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6542025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6552025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6562025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6572025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6582025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6592025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6602025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6612025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6622025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6632025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6642025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6652025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6662025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6672025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6682025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6692025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6702025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6712025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6722025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6732025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6742025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6752025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6762025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6772025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6782025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6792025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6802025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6812025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6822025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6832025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6842025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6852025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6862025-04-30T06:45:58.169ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6872025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6882025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6892025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6902025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:54590 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
6912025-04-30T06:45:58.170ZINFOtest_multiple_snapshots (dropshot): listening local_addr = [::1]:54590 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6922025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:54590 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6932025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6942025-04-30T06:45:58.170ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6952025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6962025-04-30T06:45:58.170ZWARNtest_multiple_snapshots (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6972025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 105.951\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6982025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6992025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
7002025-04-30T06:45:58.170ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
7012025-04-30T06:45:58.170ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7022025-04-30T06:45:58.172ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7032025-04-30T06:45:58.172ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
7042025-04-30T06:45:58.172ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
7052025-04-30T06:45:58.172ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
7062025-04-30T06:45:58.172ZWARNtest_multiple_snapshots (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
7072025-04-30T06:45:58.172ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 49.979\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
7082025-04-30T06:45:58.172ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
7092025-04-30T06:45:58.172ZINFOtest_multiple_snapshots (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:35:58.172522954 UTC
7102025-04-30T06:45:58.172ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
7112025-04-30T06:45:58.173ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
7122025-04-30T06:45:58.173ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
7132025-04-30T06:45:58.173ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
7142025-04-30T06:45:58.173ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
7152025-04-30T06:45:58.173ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7162025-04-30T06:45:58.173ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
7172025-04-30T06:45:58.173ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7182025-04-30T06:45:58.174ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7192025-04-30T06:45:58.174ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
7202025-04-30T06:45:58.174ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
7212025-04-30T06:45:58.174ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7222025-04-30T06:45:58.175ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7232025-04-30T06:45:58.175ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
7242025-04-30T06:45:58.175ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
7252025-04-30T06:45:58.175ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7262025-04-30T06:45:58.175ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7272025-04-30T06:45:58.175ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
7282025-04-30T06:45:58.176ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
7292025-04-30T06:45:58.176ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
7302025-04-30T06:45:58.176ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7312025-04-30T06:45:58.176ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
7322025-04-30T06:45:58.176ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7332025-04-30T06:45:58.177ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7342025-04-30T06:45:58.177ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
7352025-04-30T06:45:58.177ZWARNtest_multiple_snapshots (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
7362025-04-30T06:45:58.178ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 1.001535ms iteration = 1 status = {"error":"no blueprint"}
7372025-04-30T06:45:58.179ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
7382025-04-30T06:45:58.179ZWARNtest_multiple_snapshots (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
7392025-04-30T06:45:58.179ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 177.624\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7402025-04-30T06:45:58.179ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
7412025-04-30T06:45:58.180ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
7422025-04-30T06:45:58.180ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7432025-04-30T06:45:58.180ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
7442025-04-30T06:45:58.180ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7452025-04-30T06:45:58.181ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7462025-04-30T06:45:58.181ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
7472025-04-30T06:45:58.181ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
7482025-04-30T06:45:58.182ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7492025-04-30T06:45:58.182ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
7502025-04-30T06:45:58.182ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7512025-04-30T06:45:58.183ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7522025-04-30T06:45:58.183ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
7532025-04-30T06:45:58.183ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 145.289\u{b5}s iteration = 1 status = {"error":"task disabled"}
7542025-04-30T06:45:58.184ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
7552025-04-30T06:45:58.184ZWARNtest_multiple_snapshots (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
7562025-04-30T06:45:58.184ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 189.137\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7572025-04-30T06:45:58.185ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
7582025-04-30T06:45:58.185ZINFOtest_multiple_snapshots (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
7592025-04-30T06:45:58.185ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 258.916\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
7602025-04-30T06:45:58.185ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
7612025-04-30T06:45:58.185ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
7622025-04-30T06:45:58.185ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:45:58.185ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
7642025-04-30T06:45:58.185ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7652025-04-30T06:45:58.186ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7662025-04-30T06:45:58.186ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
7672025-04-30T06:45:58.186ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
7682025-04-30T06:45:58.186ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7692025-04-30T06:45:58.186ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
7702025-04-30T06:45:58.186ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7712025-04-30T06:45:58.187ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7722025-04-30T06:45:58.187ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7732025-04-30T06:45:58.188ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
7742025-04-30T06:45:58.188ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7752025-04-30T06:45:58.188ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7762025-04-30T06:45:58.188ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7772025-04-30T06:45:58.188ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7782025-04-30T06:45:58.189ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7792025-04-30T06:45:58.198ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7802025-04-30T06:45:58.198ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7812025-04-30T06:45:58.198ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7822025-04-30T06:45:58.198ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7832025-04-30T06:45:58.199ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7842025-04-30T06:45:58.199ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7852025-04-30T06:45:58.199ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7862025-04-30T06:45:58.199ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7872025-04-30T06:45:58.199ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7882025-04-30T06:45:58.199ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7892025-04-30T06:45:58.200ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7902025-04-30T06:45:58.200ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7912025-04-30T06:45:58.200ZINFOtest_multiple_snapshots (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7922025-04-30T06:45:58.202ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = instance_updater elapsed = 1.945076ms iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7932025-04-30T06:45:58.202ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7942025-04-30T06:45:58.202ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7952025-04-30T06:45:58.203ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7962025-04-30T06:45:58.204ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7972025-04-30T06:45:58.204ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7982025-04-30T06:45:58.204ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7992025-04-30T06:45:58.204ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
8002025-04-30T06:45:58.205ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8012025-04-30T06:45:58.206ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8022025-04-30T06:45:58.206ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
8032025-04-30T06:45:58.207ZINFOtest_multiple_snapshots (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
8042025-04-30T06:45:58.207ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
8052025-04-30T06:45:58.207ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8062025-04-30T06:45:58.207ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
8072025-04-30T06:45:58.207ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8082025-04-30T06:45:58.208ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8092025-04-30T06:45:58.210ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
8102025-04-30T06:45:58.210ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
8112025-04-30T06:45:58.210ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8122025-04-30T06:45:58.214ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8132025-04-30T06:45:58.214ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
8142025-04-30T06:45:58.214ZDEBGtest_multiple_snapshots (SagaRecoverer): listing candidate sagas for recovery
8152025-04-30T06:45:58.214ZTRACtest_multiple_snapshots (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
8162025-04-30T06:45:58.214ZDEBGtest_multiple_snapshots (SagaRecoverer): roles roles = RoleSet { roles: {} }
8172025-04-30T06:45:58.215ZDEBGtest_multiple_snapshots (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
8182025-04-30T06:45:58.215ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
8192025-04-30T06:45:58.215ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
8202025-04-30T06:45:58.215ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8212025-04-30T06:45:58.216ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8222025-04-30T06:45:58.216ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
8232025-04-30T06:45:58.217ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
8242025-04-30T06:45:58.217ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8252025-04-30T06:45:58.219ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8262025-04-30T06:45:58.219ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
8272025-04-30T06:45:58.220ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
8282025-04-30T06:45:58.220ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8292025-04-30T06:45:58.220ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8302025-04-30T06:45:58.220ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
8312025-04-30T06:45:58.221ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
8322025-04-30T06:45:58.221ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8332025-04-30T06:45:58.221ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8342025-04-30T06:45:58.222ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
8352025-04-30T06:45:58.222ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
8362025-04-30T06:45:58.222ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8372025-04-30T06:45:58.223ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8382025-04-30T06:45:58.223ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
8392025-04-30T06:45:58.223ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
8402025-04-30T06:45:58.223ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8412025-04-30T06:45:58.223ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
8422025-04-30T06:45:58.223ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8432025-04-30T06:45:58.224ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8442025-04-30T06:45:58.224ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
8452025-04-30T06:45:58.224ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
8462025-04-30T06:45:58.224ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8472025-04-30T06:45:58.225ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8482025-04-30T06:45:58.225ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
8492025-04-30T06:45:58.225ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
8502025-04-30T06:45:58.225ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
8512025-04-30T06:45:58.227ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
8522025-04-30T06:45:58.227ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
8532025-04-30T06:45:58.227ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
8542025-04-30T06:45:58.227ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8552025-04-30T06:45:58.227ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8562025-04-30T06:45:58.228ZDEBGtest_multiple_snapshots (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8572025-04-30T06:45:58.229ZDEBGtest_multiple_snapshots (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:55015/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8582025-04-30T06:45:58.231ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8592025-04-30T06:45:58.231ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8602025-04-30T06:45:58.231ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8612025-04-30T06:45:58.231ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8622025-04-30T06:45:58.231ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8632025-04-30T06:45:58.232ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:35577
8642025-04-30T06:45:58.233ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8652025-04-30T06:45:58.233ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8662025-04-30T06:45:58.233ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8672025-04-30T06:45:58.233ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8682025-04-30T06:45:58.234ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8692025-04-30T06:45:58.237ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35577 req_id = 2fdb8cd9-5d12-4699-8c2c-206481a5277f uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8702025-04-30T06:45:58.237ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: b5 c7 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:54763 req_id = 9c62b6b5-f0a7-49f8-b30a-0deee7cbb783
8712025-04-30T06:45:58.237ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46535,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54763 req_id = 9c62b6b5-f0a7-49f8-b30a-0deee7cbb783
8722025-04-30T06:45:58.237ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46535, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54763 req_id = 9c62b6b5-f0a7-49f8-b30a-0deee7cbb783
8732025-04-30T06:45:58.237ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 05 63 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 .c..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64896 req_id = 8d53a72b-20ea-4248-a9b2-589378adfe60
8742025-04-30T06:45:58.237ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1379,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64896 req_id = 8d53a72b-20ea-4248-a9b2-589378adfe60
8752025-04-30T06:45:58.238ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1379, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64896 req_id = 8d53a72b-20ea-4248-a9b2-589378adfe60
8762025-04-30T06:45:58.242ZINFOtest_multiple_snapshots (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8772025-04-30T06:45:58.243ZWARNtest_multiple_snapshots (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:45:58.243ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 73.631015ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:45:58.244ZWARNtest_multiple_snapshots (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8802025-04-30T06:45:58.244ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 71.931499ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8812025-04-30T06:45:58.259ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8822025-04-30T06:45:58.260ZINFOtest_multiple_snapshots (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:35:58.172522954 UTC pruned = {}
8832025-04-30T06:45:58.260ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 87.797301ms iteration = 1 status = {"expiration":"2025-04-30T06:35:58.172522954Z","pruned":[]}
8842025-04-30T06:45:58.260ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8852025-04-30T06:45:58.265ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 0a b8 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50279 req_id = 2fac1d0e-bd19-475c-b149-521fc58b3ff8
8862025-04-30T06:45:58.265ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2744,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50279 req_id = 2fac1d0e-bd19-475c-b149-521fc58b3ff8
8872025-04-30T06:45:58.266ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2744, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50279 req_id = 2fac1d0e-bd19-475c-b149-521fc58b3ff8
8882025-04-30T06:45:58.266ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 51 f2 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 Q............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63944 req_id = 98d5d249-e672-4681-9391-807ce2b096cb
8892025-04-30T06:45:58.266ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20978,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63944 req_id = 98d5d249-e672-4681-9391-807ce2b096cb
8902025-04-30T06:45:58.266ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20978, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63944 req_id = 98d5d249-e672-4681-9391-807ce2b096cb
8912025-04-30T06:45:58.269ZERROtest_multiple_snapshots (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8922025-04-30T06:45:58.277ZERROtest_multiple_snapshots (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8932025-04-30T06:45:58.277ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 103.021023ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8942025-04-30T06:45:58.277ZERROtest_multiple_snapshots (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8952025-04-30T06:45:58.277ZERROtest_multiple_snapshots (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:45:58.278ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 102.769372ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8972025-04-30T06:45:58.303ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 127.527581ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8982025-04-30T06:45:58.306ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8992025-04-30T06:45:58.313ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9002025-04-30T06:45:58.313ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9012025-04-30T06:45:58.313ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9022025-04-30T06:45:58.313ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9032025-04-30T06:45:58.340ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:45:58.351ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9052025-04-30T06:45:58.351ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9062025-04-30T06:45:58.351ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9072025-04-30T06:45:58.351ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9082025-04-30T06:45:58.352ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:45:58.362ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9102025-04-30T06:45:58.362ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9112025-04-30T06:45:58.362ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9122025-04-30T06:45:58.363ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9132025-04-30T06:45:58.364ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:45:58.370ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9152025-04-30T06:45:58.370ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9162025-04-30T06:45:58.370ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9172025-04-30T06:45:58.370ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9182025-04-30T06:45:58.461ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:45:58.469ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9202025-04-30T06:45:58.469ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9212025-04-30T06:45:58.469ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9222025-04-30T06:45:58.470ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9232025-04-30T06:45:58.476ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:45:58.501ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9252025-04-30T06:45:58.501ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9262025-04-30T06:45:58.501ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9272025-04-30T06:45:58.502ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9282025-04-30T06:45:58.502ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9292025-04-30T06:45:58.519ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9302025-04-30T06:45:58.519ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9312025-04-30T06:45:58.520ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9322025-04-30T06:45:58.520ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9332025-04-30T06:45:58.521ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9342025-04-30T06:45:58.521ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9352025-04-30T06:45:58.522ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9362025-04-30T06:45:58.523ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9372025-04-30T06:45:58.523ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9382025-04-30T06:45:58.524ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9392025-04-30T06:45:58.575ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9402025-04-30T06:45:58.575ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9412025-04-30T06:45:58.576ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9422025-04-30T06:45:58.623ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 408.52547ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9432025-04-30T06:45:58.653ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9442025-04-30T06:45:58.659ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9452025-04-30T06:45:58.659ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9462025-04-30T06:45:58.659ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9472025-04-30T06:45:58.659ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9482025-04-30T06:45:58.659ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9492025-04-30T06:45:58.661ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9502025-04-30T06:45:58.698ZTRACtest_multiple_snapshots (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9512025-04-30T06:45:58.698ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9522025-04-30T06:45:58.698ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9532025-04-30T06:45:58.699ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9542025-04-30T06:45:58.707ZINFOtest_multiple_snapshots (SagaRecoverer): listed in-progress sagas count = 0
9552025-04-30T06:45:58.707ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 493.72749ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9562025-04-30T06:45:58.733ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9572025-04-30T06:45:58.744ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9582025-04-30T06:45:58.745ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 518.01436ms iteration = 1 status = {"by_rx":{},"error":null}
9592025-04-30T06:45:58.796ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9602025-04-30T06:45:58.796ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9612025-04-30T06:45:58.796ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9622025-04-30T06:45:58.796ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9632025-04-30T06:45:58.797ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9642025-04-30T06:45:58.825ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9652025-04-30T06:45:58.825ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9662025-04-30T06:45:58.825ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9672025-04-30T06:45:58.829ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9682025-04-30T06:45:58.833ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9692025-04-30T06:45:58.833ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9702025-04-30T06:45:58.833ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9712025-04-30T06:45:58.833ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9722025-04-30T06:45:58.833ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9732025-04-30T06:45:58.833ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9742025-04-30T06:45:58.841ZWARNtest_multiple_snapshots (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9752025-04-30T06:45:58.841ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 665.052654ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9762025-04-30T06:45:58.855ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 633.552031ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9772025-04-30T06:45:58.861ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9782025-04-30T06:45:58.868ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9792025-04-30T06:45:58.883ZDEBGtest_multiple_snapshots (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9802025-04-30T06:45:58.883ZINFOtest_multiple_snapshots (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9812025-04-30T06:45:58.883ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 673.56256ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9822025-04-30T06:45:58.886ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9832025-04-30T06:45:58.886ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9842025-04-30T06:45:58.887ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9852025-04-30T06:45:58.888ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9862025-04-30T06:45:58.888ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 702.889992ms iteration = 1 status = {"error":"inventory collection is None"}
9872025-04-30T06:45:58.888ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 701.923417ms iteration = 1 status = {}
9882025-04-30T06:45:58.892ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 672.76991ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9892025-04-30T06:45:58.901ZWARNtest_multiple_snapshots (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
9902025-04-30T06:45:58.901ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 810.056616ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
9912025-04-30T06:45:58.902ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9922025-04-30T06:45:58.902ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9932025-04-30T06:45:58.903ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 45 e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 E............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53011 req_id = cdad2cda-5e36-46c5-a472-3d02f14ddbef
9942025-04-30T06:45:58.903ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17888,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53011 req_id = cdad2cda-5e36-46c5-a472-3d02f14ddbef
9952025-04-30T06:45:58.904ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17888, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53011 req_id = cdad2cda-5e36-46c5-a472-3d02f14ddbef
9962025-04-30T06:45:58.905ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9972025-04-30T06:45:58.906ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 725.896512ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9982025-04-30T06:45:58.906ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9992025-04-30T06:45:58.906ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10002025-04-30T06:45:58.907ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10012025-04-30T06:45:58.908ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10022025-04-30T06:45:58.908ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10032025-04-30T06:45:58.909ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10042025-04-30T06:45:58.919ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10052025-04-30T06:45:58.920ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10062025-04-30T06:45:58.920ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10072025-04-30T06:45:58.920ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10082025-04-30T06:45:58.922ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10092025-04-30T06:45:58.926ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_replacement elapsed = 737.510211ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10102025-04-30T06:45:58.979ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 754.537406ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
10112025-04-30T06:45:58.996ZTRACtest_multiple_snapshots (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10122025-04-30T06:45:58.996ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10132025-04-30T06:45:58.996ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10142025-04-30T06:45:58.997ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10152025-04-30T06:45:58.997ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 799.061438ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10162025-04-30T06:45:59.066ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10172025-04-30T06:45:59.097ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10182025-04-30T06:45:59.098ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10192025-04-30T06:45:59.098ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10202025-04-30T06:45:59.099ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10212025-04-30T06:45:59.100ZINFOtest_multiple_snapshots (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10222025-04-30T06:45:59.100ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 901.134783ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10232025-04-30T06:45:59.100ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10242025-04-30T06:45:59.108ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10252025-04-30T06:45:59.108ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10262025-04-30T06:45:59.108ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10272025-04-30T06:45:59.109ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10282025-04-30T06:45:59.109ZWARNtest_multiple_snapshots (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10292025-04-30T06:45:59.109ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 938.786211ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10302025-04-30T06:45:59.109ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10312025-04-30T06:45:59.109ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10322025-04-30T06:45:59.110ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10332025-04-30T06:45:59.110ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10342025-04-30T06:45:59.110ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10352025-04-30T06:45:59.111ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10362025-04-30T06:45:59.111ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10372025-04-30T06:45:59.111ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10382025-04-30T06:45:59.112ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10392025-04-30T06:45:59.112ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 875453 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35577 req_id = 2fdb8cd9-5d12-4699-8c2c-206481a5277f response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10402025-04-30T06:45:59.114ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10412025-04-30T06:45:59.138ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10422025-04-30T06:45:59.138ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10432025-04-30T06:45:59.138ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10442025-04-30T06:45:59.139ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10452025-04-30T06:45:59.141ZDEBGtest_multiple_snapshots (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:55015/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "2fdb8cd9-5d12-4699-8c2c-206481a5277f", "date": "Wed, 30 Apr 2025 06:45:58 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10462025-04-30T06:45:59.141ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_crucible_pantry
10472025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10482025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10492025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10512025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10522025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10532025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10542025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10552025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10562025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10572025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): registered endpoint kind = pantry local_addr = [::1]:64080 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10582025-04-30T06:45:59.144ZINFOtest_multiple_snapshots (dropshot): listening kind = pantry local_addr = [::1]:64080 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10592025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:64080 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10602025-04-30T06:45:59.144ZINFOtest_multiple_snapshots (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:64080 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10612025-04-30T06:45:59.144ZDEBGtest_multiple_snapshots (init_with_steps): Running step populate_internal_dns
10622025-04-30T06:45:59.145ZDEBGtest_multiple_snapshots: Populating Internal DNS
10632025-04-30T06:45:59.270ZINFOtest_multiple_snapshots: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:45:59.269771671Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "3ae6adf1-675a-4c6a-a06e-473e2878df6e.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "f16209cb-89f1-42df-855d-4fa425ceb7fd.host": [ Aaaa( ::1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "61dcdbf9-3b41-4769-bc9c-738fb2f99c99.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 35840, target: "876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36409, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 55912, target: "f16209cb-89f1-42df-855d-4fa425ceb7fd.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 42239, target: "75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "61dcdbf9-3b41-4769-bc9c-738fb2f99c99.host": [ Aaaa( ::1, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 55875, target: "61dcdbf9-3b41-4769-bc9c-738fb2f99c99.host.control-plane.oxide.internal", }, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 45115, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "61dcdbf9-3b41-4769-bc9c-738fb2f99c99.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "61dcdbf9-3b41-4769-bc9c-738fb2f99c99.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 48945, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "75069e0e-8b61-4a18-8a45-042e87d80ea0.host": [ Aaaa( ::1, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 55015, target: "3ae6adf1-675a-4c6a-a06e-473e2878df6e.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 59043, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "6229309a-478e-474f-811a-5c520b1df45a.host": [ Aaaa( ::1, ), ], "876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host": [ Aaaa( ::1, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 64080, target: "6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal", }, ), ], }, }, ], }
10642025-04-30T06:45:59.270ZDEBGtest_multiple_snapshots: client request body = Some(Body) method = PUT uri = http://[::1]:35840/config
10652025-04-30T06:45:59.271ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.05510588s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10662025-04-30T06:45:59.273ZINFOtest_multiple_snapshots (http): accepted connection kind = dns local_addr = [::1]:35840 remote_addr = [::1]:52480
10672025-04-30T06:45:59.275ZTRACtest_multiple_snapshots (http): incoming request kind = dns local_addr = [::1]:35840 method = PUT remote_addr = [::1]:52480 req_id = 104a41a5-11e9-4f59-bc01-8a8fb0cecc60 uri = /config
10682025-04-30T06:45:59.276ZINFOtest_multiple_snapshots (store): attempting generation update new_generation = 1 req_id = 104a41a5-11e9-4f59-bc01-8a8fb0cecc60
10692025-04-30T06:45:59.276ZINFOtest_multiple_snapshots (store): pruning trees for generations newer than 0
10702025-04-30T06:45:59.276ZDEBGtest_multiple_snapshots (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10712025-04-30T06:45:59.277ZDEBGtest_multiple_snapshots (store): updating current config
10722025-04-30T06:45:59.277ZDEBGtest_multiple_snapshots (store): flushing default tree
10732025-04-30T06:45:59.277ZINFOtest_multiple_snapshots (store): pruning trees for generations older than 1 keep = 3
10742025-04-30T06:45:59.277ZINFOtest_multiple_snapshots (store): updated generation new_generation = 1 req_id = 104a41a5-11e9-4f59-bc01-8a8fb0cecc60
10752025-04-30T06:45:59.277ZINFOtest_multiple_snapshots (http): request completed kind = dns latency_us = 2410 local_addr = [::1]:35840 method = PUT remote_addr = [::1]:52480 req_id = 104a41a5-11e9-4f59-bc01-8a8fb0cecc60 response_code = 204 uri = /config
10762025-04-30T06:45:59.278ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10772025-04-30T06:45:59.278ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10782025-04-30T06:45:59.278ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10792025-04-30T06:45:59.278ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10802025-04-30T06:45:59.279ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10812025-04-30T06:45:59.281ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10822025-04-30T06:45:59.287ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10832025-04-30T06:45:59.287ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10842025-04-30T06:45:59.287ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10852025-04-30T06:45:59.288ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10862025-04-30T06:45:59.291ZTRACtest_multiple_snapshots (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
10872025-04-30T06:45:59.291ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.065765619s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
10882025-04-30T06:45:59.291ZDEBGtest_multiple_snapshots: client response result = Ok(Response { url: "http://[::1]:35840/config", status: 204, headers: {"x-request-id": "104a41a5-11e9-4f59-bc01-8a8fb0cecc60", "date": "Wed, 30 Apr 2025 06:45:59 GMT"} })
10892025-04-30T06:45:59.292ZDEBGtest_multiple_snapshots (init_with_steps): Running step configure_sled_agents
10902025-04-30T06:45:59.393ZDEBGtest_multiple_snapshots: client request body = Some(Body) method = PUT uri = http://[::1]:54590/omicron-config
10912025-04-30T06:45:59.394ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:50287 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10922025-04-30T06:45:59.395ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:50287 req_id = 5490874b-eed2-4c26-b60a-ad49313f3148 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10932025-04-30T06:45:59.403ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 8657 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:50287 req_id = 5490874b-eed2-4c26-b60a-ad49313f3148 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10942025-04-30T06:45:59.404ZDEBGtest_multiple_snapshots: client response result = Ok(Response { url: "http://[::1]:54590/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5490874b-eed2-4c26-b60a-ad49313f3148", "content-length": "26", "date": "Wed, 30 Apr 2025 06:45:59 GMT"} })
10952025-04-30T06:45:59.405ZDEBGtest_multiple_snapshots: client request body = Some(Body) method = PUT uri = http://[::1]:54590/network-bootstore-config
10962025-04-30T06:45:59.410ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:50287 req_id = c43190a3-f642-4ee7-b703-603396e6195b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10972025-04-30T06:45:59.413ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2527 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:50287 req_id = c43190a3-f642-4ee7-b703-603396e6195b response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10982025-04-30T06:45:59.414ZDEBGtest_multiple_snapshots: client response result = Ok(Response { url: "http://[::1]:54590/network-bootstore-config", status: 204, headers: {"x-request-id": "c43190a3-f642-4ee7-b703-603396e6195b", "date": "Wed, 30 Apr 2025 06:45:59 GMT"} })
10992025-04-30T06:45:59.416ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_nexus_external
11002025-04-30T06:45:59.416ZDEBGtest_multiple_snapshots: Starting Nexus (external API)
11012025-04-30T06:46:02.518ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
11022025-04-30T06:46:02.519ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11032025-04-30T06:46:02.519ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11042025-04-30T06:46:02.519ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
11052025-04-30T06:46:02.520ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11062025-04-30T06:46:02.531ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11072025-04-30T06:46:02.540ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11082025-04-30T06:46:02.540ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11092025-04-30T06:46:02.540ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11102025-04-30T06:46:02.541ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11112025-04-30T06:46:02.541ZWARNtest_multiple_snapshots (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11122025-04-30T06:46:02.542ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.368592274s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11132025-04-30T06:46:02.592ZINFOtest_multiple_snapshots (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11142025-04-30T06:46:02.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:54590/vpc-routes
11152025-04-30T06:46:02.593ZTRACtest_multiple_snapshots (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11162025-04-30T06:46:02.593ZINFOtest_multiple_snapshots (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11172025-04-30T06:46:02.593ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.391218015s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11182025-04-30T06:46:02.594ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11192025-04-30T06:46:02.594ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11202025-04-30T06:46:02.594ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11212025-04-30T06:46:02.594ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11222025-04-30T06:46:02.596ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11232025-04-30T06:46:02.596ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.375638093s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11242025-04-30T06:46:02.600ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:34834 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11252025-04-30T06:46:02.601ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11262025-04-30T06:46:02.601ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11272025-04-30T06:46:02.601ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11282025-04-30T06:46:02.602ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11292025-04-30T06:46:02.692ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11302025-04-30T06:46:02.692ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11312025-04-30T06:46:02.692ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11322025-04-30T06:46:02.692ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11332025-04-30T06:46:02.695ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11342025-04-30T06:46:02.697ZINFOtest_multiple_snapshots (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11352025-04-30T06:46:02.697ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.515494057s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11362025-04-30T06:46:02.700ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11372025-04-30T06:46:02.706ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11382025-04-30T06:46:02.706ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11392025-04-30T06:46:02.706ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11402025-04-30T06:46:02.706ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11412025-04-30T06:46:02.706ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11422025-04-30T06:46:02.706ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11432025-04-30T06:46:02.707ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:34834 req_id = 9646c43a-f263-427c-bff0-927e28f62276 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11442025-04-30T06:46:02.717ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 9225 local_addr = [::1]:54590 method = GET remote_addr = [::1]:34834 req_id = 9646c43a-f263-427c-bff0-927e28f62276 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11452025-04-30T06:46:02.717ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11462025-04-30T06:46:02.717ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11472025-04-30T06:46:02.718ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11482025-04-30T06:46:02.718ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11492025-04-30T06:46:02.718ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11502025-04-30T06:46:02.718ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11512025-04-30T06:46:02.720ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11522025-04-30T06:46:02.725ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11532025-04-30T06:46:02.725ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11542025-04-30T06:46:02.725ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11552025-04-30T06:46:02.725ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11562025-04-30T06:46:02.726ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11572025-04-30T06:46:02.732ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11582025-04-30T06:46:02.732ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11592025-04-30T06:46:02.732ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11602025-04-30T06:46:02.733ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11612025-04-30T06:46:02.741ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11622025-04-30T06:46:02.748ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11632025-04-30T06:46:02.748ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11642025-04-30T06:46:02.748ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
11652025-04-30T06:46:02.749ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11662025-04-30T06:46:02.751ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:54590/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9646c43a-f263-427c-bff0-927e28f62276", "content-length": "2", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
11672025-04-30T06:46:02.752ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11682025-04-30T06:46:02.752ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11692025-04-30T06:46:02.754ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11702025-04-30T06:46:02.756ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11712025-04-30T06:46:02.757ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11722025-04-30T06:46:02.757ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11732025-04-30T06:46:02.757ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11742025-04-30T06:46:02.758ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11752025-04-30T06:46:02.767ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 54 2a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 T*..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62584 req_id = 6b2909d7-0dc5-4652-9075-9511411a5713
11762025-04-30T06:46:02.767ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21546,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62584 req_id = 6b2909d7-0dc5-4652-9075-9511411a5713
11772025-04-30T06:46:02.767ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
11782025-04-30T06:46:02.768ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11792025-04-30T06:46:02.768ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62584 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6b2909d7-0dc5-4652-9075-9511411a5713
11802025-04-30T06:46:02.769ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11812025-04-30T06:46:02.769ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
11822025-04-30T06:46:02.782ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11832025-04-30T06:46:02.783ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4427, tv_nsec: 582390392 } })
11842025-04-30T06:46:02.783ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
11852025-04-30T06:46:02.794ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11862025-04-30T06:46:02.798ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11872025-04-30T06:46:02.798ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11882025-04-30T06:46:02.798ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11892025-04-30T06:46:02.799ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11902025-04-30T06:46:02.802ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11912025-04-30T06:46:02.812ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11922025-04-30T06:46:02.812ZTRACtest_multiple_snapshots (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11932025-04-30T06:46:02.812ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11942025-04-30T06:46:02.812ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11952025-04-30T06:46:02.817ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11962025-04-30T06:46:02.817ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11972025-04-30T06:46:02.817ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11982025-04-30T06:46:02.819ZINFOtest_multiple_snapshots (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
11992025-04-30T06:46:02.819ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12002025-04-30T06:46:02.820ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
12012025-04-30T06:46:02.820ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12022025-04-30T06:46:02.821ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 29 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e )...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58795 req_id = 87bd2596-74d6-4b07-84cd-2227b61f5500
12032025-04-30T06:46:02.821ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10521,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58795 req_id = 87bd2596-74d6-4b07-84cd-2227b61f5500
12042025-04-30T06:46:02.823ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
12052025-04-30T06:46:02.823ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58795 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 87bd2596-74d6-4b07-84cd-2227b61f5500
12062025-04-30T06:46:02.825ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12072025-04-30T06:46:02.825ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12082025-04-30T06:46:02.825ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12092025-04-30T06:46:02.826ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12102025-04-30T06:46:02.828ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:54590/artifacts-config
12112025-04-30T06:46:02.909ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
12122025-04-30T06:46:02.909ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:36990 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12132025-04-30T06:46:02.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:54590/v2p
12142025-04-30T06:46:02.978ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12152025-04-30T06:46:02.978ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12162025-04-30T06:46:02.978ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12172025-04-30T06:46:02.979ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12182025-04-30T06:46:02.981ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12192025-04-30T06:46:02.981ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12202025-04-30T06:46:02.981ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12212025-04-30T06:46:02.981ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
12222025-04-30T06:46:02.982ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12232025-04-30T06:46:02.983ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:36990 req_id = fe4f0cfc-1121-4d97-8df1-92aa8d6714d8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12242025-04-30T06:46:02.983ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:63467 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12252025-04-30T06:46:02.983ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12262025-04-30T06:46:02.991ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12272025-04-30T06:46:02.991ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12282025-04-30T06:46:02.991ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12292025-04-30T06:46:02.992ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12302025-04-30T06:46:02.993ZINFOtest_multiple_snapshots (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpcGE6OU/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12312025-04-30T06:46:02.993ZDEBGtest_multiple_snapshots (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpcGE6OU/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12322025-04-30T06:46:02.993ZINFOtest_multiple_snapshots (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpLSfmRQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12332025-04-30T06:46:02.993ZDEBGtest_multiple_snapshots (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpLSfmRQ/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12342025-04-30T06:46:02.993ZINFOtest_multiple_snapshots (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpcGE6OU/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12352025-04-30T06:46:02.995ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:63467 req_id = 87283860-ab99-4a6a-b582-baabc71f3df0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12362025-04-30T06:46:02.996ZINFOtest_multiple_snapshots (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpLSfmRQ/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12372025-04-30T06:46:03.002ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4b 40 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c K@..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54787 req_id = 0dd63aff-7a2e-43ef-9371-8e2d8c05202c
12382025-04-30T06:46:03.002ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19264,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54787 req_id = 0dd63aff-7a2e-43ef-9371-8e2d8c05202c
12392025-04-30T06:46:03.003ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
12402025-04-30T06:46:03.003ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19264, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54787 req_id = 0dd63aff-7a2e-43ef-9371-8e2d8c05202c
12412025-04-30T06:46:03.004ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 9112 local_addr = [::1]:54590 method = GET remote_addr = [::1]:63467 req_id = 87283860-ab99-4a6a-b582-baabc71f3df0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12422025-04-30T06:46:03.006ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
12432025-04-30T06:46:03.006ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
12442025-04-30T06:46:03.006ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
12452025-04-30T06:46:03.006ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:54590/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "87283860-ab99-4a6a-b582-baabc71f3df0", "content-length": "2", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
12462025-04-30T06:46:03.007ZINFOtest_multiple_snapshots (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12472025-04-30T06:46:03.007ZINFOtest_multiple_snapshots (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12482025-04-30T06:46:03.007ZINFOtest_multiple_snapshots (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12492025-04-30T06:46:03.008ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 4.820715219s iteration = 1 status = {}
12502025-04-30T06:46:03.008ZINFOtest_multiple_snapshots (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12512025-04-30T06:46:03.009ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 25630 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:36990 req_id = fe4f0cfc-1121-4d97-8df1-92aa8d6714d8 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12522025-04-30T06:46:03.009ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12532025-04-30T06:46:03.009ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12542025-04-30T06:46:03.010ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12552025-04-30T06:46:03.013ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:54590/artifacts-config", status: 204, headers: {"x-request-id": "fe4f0cfc-1121-4d97-8df1-92aa8d6714d8", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
12562025-04-30T06:46:03.014ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:54590/artifacts
12572025-04-30T06:46:03.015ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:36990 req_id = 2d8643a2-ab67-414b-9eec-97beb29a2c78 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12582025-04-30T06:46:03.016ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 718 local_addr = [::1]:54590 method = GET remote_addr = [::1]:36990 req_id = 2d8643a2-ab67-414b-9eec-97beb29a2c78 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12592025-04-30T06:46:03.016ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12602025-04-30T06:46:03.016ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12612025-04-30T06:46:03.017ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12622025-04-30T06:46:03.020ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:54590/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8643a2-ab67-414b-9eec-97beb29a2c78", "content-length": "26", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
12632025-04-30T06:46:03.021ZINFOtest_multiple_snapshots (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:54590
12642025-04-30T06:46:03.024ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 4.801111249s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:46:03.013946794Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:46:03.021012291Z"}]}
12652025-04-30T06:46:03.029ZINFOtest_multiple_snapshots (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12662025-04-30T06:46:03.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:54590/eip-gateways
12672025-04-30T06:46:03.029ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12682025-04-30T06:46:03.029ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12692025-04-30T06:46:03.031ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12702025-04-30T06:46:03.032ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:34834 req_id = 1158a97b-a8f4-4d21-a357-3447206073e3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12712025-04-30T06:46:03.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab30a2b-441d-47b1-871b-aed40d51df88", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
12722025-04-30T06:46:03.036ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
12732025-04-30T06:46:03.037ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 4866 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:34834 req_id = 1158a97b-a8f4-4d21-a357-3447206073e3 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12742025-04-30T06:46:03.038ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:54590/eip-gateways", status: 204, headers: {"x-request-id": "1158a97b-a8f4-4d21-a357-3447206073e3", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
12752025-04-30T06:46:03.042ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.835954623s iteration = 1 status = {}
12762025-04-30T06:46:03.044ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12772025-04-30T06:46:03.044ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12782025-04-30T06:46:03.044ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12792025-04-30T06:46:03.044ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12802025-04-30T06:46:03.046ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12812025-04-30T06:46:03.047ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12822025-04-30T06:46:03.054ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12832025-04-30T06:46:03.054ZDEBGtest_multiple_snapshots (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12842025-04-30T06:46:03.054ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12852025-04-30T06:46:03.055ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
12862025-04-30T06:46:03.055ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12872025-04-30T06:46:03.060ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12882025-04-30T06:46:03.060ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12892025-04-30T06:46:03.060ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12902025-04-30T06:46:03.060ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12912025-04-30T06:46:03.065ZINFOtest_multiple_snapshots (InternalApi): Using static configuration for external switchports
12922025-04-30T06:46:03.065ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12932025-04-30T06:46:03.065ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
12942025-04-30T06:46:03.066ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12952025-04-30T06:46:03.067ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:46:03.067ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:46:03.067ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:46:03.068ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:46:03.091ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:46:03.100ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:46:03.100ZTRACtest_multiple_snapshots (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:46:03.100ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:46:03.101ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:46:03.102ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:46:03.103ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de3eace7-a751-4456-b2be-461e847a1e01", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
13062025-04-30T06:46:03.104ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
13072025-04-30T06:46:03.110ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13082025-04-30T06:46:03.110ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13092025-04-30T06:46:03.111ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13102025-04-30T06:46:03.141ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13112025-04-30T06:46:03.141ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13122025-04-30T06:46:03.142ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13132025-04-30T06:46:03.145ZINFOtest_multiple_snapshots (InternalApi): Recording Rack Network Configuration
13142025-04-30T06:46:03.145ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13152025-04-30T06:46:03.145ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
13162025-04-30T06:46:03.146ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13172025-04-30T06:46:03.148ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:46:03.148ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:46:03.149ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:46:03.155ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:46:03.155ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:46:03.155ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:46:03.155ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:46:03.156ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:46:03.158ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:46:03.158ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:46:03.159ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:46:03.159ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:46:03.165ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:46:03.165ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:46:03.165ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:46:03.166ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:46:03.177ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:46:03.195ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:46:03.195ZTRACtest_multiple_snapshots (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:46:03.195ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:46:03.195ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:46:03.195ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:46:03.195ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:46:03.196ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:46:03.210ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13422025-04-30T06:46:03.210ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13432025-04-30T06:46:03.210ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442025-04-30T06:46:03.211ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452025-04-30T06:46:03.216ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13462025-04-30T06:46:03.216ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13472025-04-30T06:46:03.216ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13482025-04-30T06:46:03.216ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13492025-04-30T06:46:03.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08424239-9b23-43db-9560-bbbf978e2aec", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
13502025-04-30T06:46:03.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
13512025-04-30T06:46:03.229ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13522025-04-30T06:46:03.239ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13532025-04-30T06:46:03.239ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13542025-04-30T06:46:03.239ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13552025-04-30T06:46:03.240ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13562025-04-30T06:46:03.251ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13572025-04-30T06:46:03.251ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13582025-04-30T06:46:03.253ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13592025-04-30T06:46:03.266ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13602025-04-30T06:46:03.266ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13612025-04-30T06:46:03.268ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13622025-04-30T06:46:03.269ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13632025-04-30T06:46:03.269ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13642025-04-30T06:46:03.269ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13652025-04-30T06:46:03.269ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
13662025-04-30T06:46:03.270ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13672025-04-30T06:46:03.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ffbc9f-6687-49ae-b7e4-5f6fe7638428", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
13682025-04-30T06:46:03.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
13692025-04-30T06:46:03.281ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13702025-04-30T06:46:03.281ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13712025-04-30T06:46:03.282ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13722025-04-30T06:46:03.287ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13732025-04-30T06:46:03.291ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13742025-04-30T06:46:03.292ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13752025-04-30T06:46:03.292ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
13762025-04-30T06:46:03.292ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13772025-04-30T06:46:03.293ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13782025-04-30T06:46:03.293ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13792025-04-30T06:46:03.293ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13802025-04-30T06:46:03.299ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") }
13812025-04-30T06:46:03.299ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13822025-04-30T06:46:03.299ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13832025-04-30T06:46:03.299ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
13842025-04-30T06:46:03.299ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13852025-04-30T06:46:03.300ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13862025-04-30T06:46:03.300ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13872025-04-30T06:46:03.300ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13882025-04-30T06:46:03.300ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13892025-04-30T06:46:03.301ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13902025-04-30T06:46:03.310ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13912025-04-30T06:46:03.316ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13922025-04-30T06:46:03.316ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13932025-04-30T06:46:03.316ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
13942025-04-30T06:46:03.317ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13952025-04-30T06:46:03.318ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13962025-04-30T06:46:03.318ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13972025-04-30T06:46:03.318ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13982025-04-30T06:46:03.318ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13992025-04-30T06:46:03.327ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14002025-04-30T06:46:03.327ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14012025-04-30T06:46:03.327ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14022025-04-30T06:46:03.328ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14032025-04-30T06:46:03.331ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") }
14042025-04-30T06:46:03.331ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14052025-04-30T06:46:03.331ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14062025-04-30T06:46:03.331ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
14072025-04-30T06:46:03.331ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14082025-04-30T06:46:03.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e1ace5-f697-4a56-8864-5af6d2d28a22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14092025-04-30T06:46:03.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
14102025-04-30T06:46:03.332ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14112025-04-30T06:46:03.341ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14122025-04-30T06:46:03.341ZTRACtest_multiple_snapshots (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14132025-04-30T06:46:03.341ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14142025-04-30T06:46:03.341ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14152025-04-30T06:46:03.341ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14162025-04-30T06:46:03.341ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14172025-04-30T06:46:03.342ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14182025-04-30T06:46:03.344ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14192025-04-30T06:46:03.344ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14202025-04-30T06:46:03.344ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14212025-04-30T06:46:03.345ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14222025-04-30T06:46:03.345ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14232025-04-30T06:46:03.351ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14242025-04-30T06:46:03.355ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14252025-04-30T06:46:03.355ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14262025-04-30T06:46:03.355ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14272025-04-30T06:46:03.361ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14282025-04-30T06:46:03.364ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14292025-04-30T06:46:03.381ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14302025-04-30T06:46:03.381ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14312025-04-30T06:46:03.381ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14322025-04-30T06:46:03.383ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14332025-04-30T06:46:03.383ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") }
14342025-04-30T06:46:03.384ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14352025-04-30T06:46:03.384ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14362025-04-30T06:46:03.384ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
14372025-04-30T06:46:03.385ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14382025-04-30T06:46:03.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fff7b32-8515-4eca-a043-0ac6ad678c2c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14392025-04-30T06:46:03.387ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
14402025-04-30T06:46:03.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32af1016-1c9c-440f-bf2e-58cd4f1c185c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14412025-04-30T06:46:03.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
14422025-04-30T06:46:03.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17fd79bd-1108-4b91-b5da-df3715f64474", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14432025-04-30T06:46:03.400ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
14442025-04-30T06:46:03.403ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14452025-04-30T06:46:03.406ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: e696f52c-7341-468d-b235-6f9f72a64300, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14462025-04-30T06:46:03.410ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7112eb-66d9-446a-9702-5b7ec218dcd9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14472025-04-30T06:46:03.410ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
14482025-04-30T06:46:03.412ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
14492025-04-30T06:46:03.412ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14502025-04-30T06:46:03.412ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14512025-04-30T06:46:03.412ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14522025-04-30T06:46:03.412ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14532025-04-30T06:46:03.412ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14542025-04-30T06:46:03.414ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14552025-04-30T06:46:03.414ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14562025-04-30T06:46:03.414ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14572025-04-30T06:46:03.414ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
14582025-04-30T06:46:03.414ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14592025-04-30T06:46:03.426ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14602025-04-30T06:46:03.433ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14612025-04-30T06:46:03.433ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14622025-04-30T06:46:03.434ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14632025-04-30T06:46:03.436ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14642025-04-30T06:46:03.438ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14652025-04-30T06:46:03.443ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14662025-04-30T06:46:03.450ZTRACtest_multiple_snapshots (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14672025-04-30T06:46:03.450ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14682025-04-30T06:46:03.450ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14692025-04-30T06:46:03.450ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14702025-04-30T06:46:03.454ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b199bf2-3f57-4898-9389-67754eb2348e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14712025-04-30T06:46:03.455ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
14722025-04-30T06:46:03.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7d7dc4-f228-4af6-a3b0-55ea3d1b5c4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14732025-04-30T06:46:03.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
14742025-04-30T06:46:03.471ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07830323-a942-4780-8d3c-1e0c4c3a1679", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14752025-04-30T06:46:03.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
14762025-04-30T06:46:03.473ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe1f880-7bb7-4d53-870e-4c532fdaecbe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14772025-04-30T06:46:03.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:46:03.475ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c9bc6a7-8dd8-4e3d-86fa-870e56b17209", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14792025-04-30T06:46:03.476ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:46:03.478ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d39d6d-0e5a-444f-bbe6-1c3ab5a6cbc0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14812025-04-30T06:46:03.478ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:46:03.481ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c8adb88-65d2-4d40-ada5-922d758c0476", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14832025-04-30T06:46:03.482ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:46:03.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae63fc9b-fa1d-4fd7-adef-63db3a0b48c2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14852025-04-30T06:46:03.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:46:03.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d453e0c7-21af-4303-92fb-de62ebfe19da", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14872025-04-30T06:46:03.499ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:46:03.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83d0c995-b0d0-4443-96e1-078844ae5f59", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14892025-04-30T06:46:03.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
14902025-04-30T06:46:03.505ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23b3731a-6a2e-4fcd-9fbf-28a4d4360baf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14912025-04-30T06:46:03.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
14922025-04-30T06:46:03.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2155b78-2bfe-4c7a-9a44-23c015aa5b37", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14932025-04-30T06:46:03.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
14942025-04-30T06:46:03.509ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3655aef1-a037-4b41-9236-d8f2ec78473d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14952025-04-30T06:46:03.509ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
14962025-04-30T06:46:03.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a6a1f9d-9fa3-4144-87cb-b05bbbb7af38", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14972025-04-30T06:46:03.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
14982025-04-30T06:46:03.517ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8d68066-352f-46b8-9778-3bae648c30a1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
14992025-04-30T06:46:03.518ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
15002025-04-30T06:46:03.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "584d311c-dec5-4c92-a6f5-7047cb6a5f07", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15012025-04-30T06:46:03.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
15022025-04-30T06:46:03.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e576f91a-efed-4efa-8b3e-246aa2649d98", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15032025-04-30T06:46:03.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
15042025-04-30T06:46:03.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "757be39a-0739-4f2a-8d72-1f8255d6d75a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15052025-04-30T06:46:03.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
15062025-04-30T06:46:03.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4f804e2-b2da-4d6f-99b8-0775cbe7e2d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15072025-04-30T06:46:03.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
15082025-04-30T06:46:03.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05051276-b1c1-4470-90b2-699340460490", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15092025-04-30T06:46:03.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
15102025-04-30T06:46:03.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90443bed-0306-4c7f-922b-1eedb8a61194", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15112025-04-30T06:46:03.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
15122025-04-30T06:46:03.537ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7ec8971-dfc7-4c4d-af97-da36fc2dd9b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15132025-04-30T06:46:03.537ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
15142025-04-30T06:46:03.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f0c6aa9-ccdc-4d98-a0de-967f763ca6dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15152025-04-30T06:46:03.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
15162025-04-30T06:46:03.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe86b87b-73ab-4309-b335-493253066bee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15172025-04-30T06:46:03.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
15182025-04-30T06:46:03.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3cd53ae-5960-4464-a4b1-b7aa68256f5d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15192025-04-30T06:46:03.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
15202025-04-30T06:46:03.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97e3f6c-97a3-408e-82b8-fcc54470efe7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15212025-04-30T06:46:03.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
15222025-04-30T06:46:03.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed5e429-6e53-48b4-884c-d68ee877410e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15232025-04-30T06:46:03.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
15242025-04-30T06:46:03.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4297c072-a99b-4a26-aef3-5409c5da6455", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15252025-04-30T06:46:03.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
15262025-04-30T06:46:03.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e216605c-58fc-42a7-a840-20479bb188c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15272025-04-30T06:46:03.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
15282025-04-30T06:46:03.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c93e9be-e1d2-4196-bd5d-685bdb8d7c88", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15292025-04-30T06:46:03.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
15302025-04-30T06:46:03.553ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d58011f-34dd-4293-a4a6-fbd6ef3d2a19", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15312025-04-30T06:46:03.554ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
15322025-04-30T06:46:03.559ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d7c5ef5-3ada-47d0-978d-dea749e49cda", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15332025-04-30T06:46:03.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
15342025-04-30T06:46:03.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc61474c-a597-4896-bd47-195e2a039f78", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15352025-04-30T06:46:03.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
15362025-04-30T06:46:03.562ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87bfbdb1-cad5-4956-9c0f-56df624660f3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15372025-04-30T06:46:03.562ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
15382025-04-30T06:46:03.569ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc7c6361-3f00-42ec-a112-fd885345cf29", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15392025-04-30T06:46:03.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
15402025-04-30T06:46:03.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "190dcd7a-4a62-446c-8828-bc46c206f58d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:02 GMT"} })
15412025-04-30T06:46:03.571ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15422025-04-30T06:46:03.571ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15432025-04-30T06:46:03.571ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15442025-04-30T06:46:03.571ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15452025-04-30T06:46:03.573ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15462025-04-30T06:46:03.590ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15472025-04-30T06:46:03.596ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15482025-04-30T06:46:03.596ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15492025-04-30T06:46:03.596ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15502025-04-30T06:46:03.596ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15512025-04-30T06:46:03.596ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15522025-04-30T06:46:03.596ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15532025-04-30T06:46:03.606ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-04-30T06:46:03.615ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15552025-04-30T06:46:03.615ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15562025-04-30T06:46:03.615ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15572025-04-30T06:46:03.616ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15582025-04-30T06:46:03.616ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15592025-04-30T06:46:03.616ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15602025-04-30T06:46:03.616ZINFOtest_multiple_snapshots (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15612025-04-30T06:46:03.617ZDEBGtest_multiple_snapshots (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.412421416s
15622025-04-30T06:46:03.617ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.412781243s iteration = 1 status = {}
15632025-04-30T06:46:03.697ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
15642025-04-30T06:46:03.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
15652025-04-30T06:46:03.698ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:45307 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15662025-04-30T06:46:03.699ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:45307 req_id = ead7f76c-a09d-49c0-888b-67f87a4b5ae3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15672025-04-30T06:46:03.700ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 618 local_addr = [::1]:54590 method = GET remote_addr = [::1]:45307 req_id = ead7f76c-a09d-49c0-888b-67f87a4b5ae3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15682025-04-30T06:46:03.700ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead7f76c-a09d-49c0-888b-67f87a4b5ae3", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:46:03 GMT"} })
15692025-04-30T06:46:03.703ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15702025-04-30T06:46:03.703ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15712025-04-30T06:46:03.703ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
15722025-04-30T06:46:03.705ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15732025-04-30T06:46:03.705ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15742025-04-30T06:46:03.705ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15752025-04-30T06:46:03.705ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15762025-04-30T06:46:03.706ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15772025-04-30T06:46:03.713ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15782025-04-30T06:46:03.717ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15792025-04-30T06:46:03.717ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15802025-04-30T06:46:03.717ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15812025-04-30T06:46:03.718ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15822025-04-30T06:46:03.720ZINFOtest_multiple_snapshots (InternalApi): inserted blueprint blueprint_id = cdd0a56d-5762-4e54-9fec-d60ce87c4ab2
15832025-04-30T06:46:03.720ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15842025-04-30T06:46:03.720ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15852025-04-30T06:46:03.720ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15862025-04-30T06:46:03.720ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
15872025-04-30T06:46:03.722ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15882025-04-30T06:46:03.726ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15892025-04-30T06:46:03.735ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15902025-04-30T06:46:03.819ZINFOtest_multiple_snapshots (InternalApi): Inserted networking records for nexus service
15912025-04-30T06:46:03.819ZINFOtest_multiple_snapshots (InternalApi): No networking records needed for clickhouse service
15922025-04-30T06:46:03.819ZINFOtest_multiple_snapshots (InternalApi): No networking records needed for crucible_pantry service
15932025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): Inserted networking records for external_dns service
15942025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): No networking records needed for internal_dns service
15952025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): No networking records needed for cockroach_db service
15962025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): Inserted service networking records
15972025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): Inserted physical disks
15982025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): Inserted zpools
15992025-04-30T06:46:03.859ZINFOtest_multiple_snapshots (InternalApi): Inserted datasets
16002025-04-30T06:46:03.878ZINFOtest_multiple_snapshots (InternalApi): Populated DNS tables for internal DNS
16012025-04-30T06:46:03.883ZINFOtest_multiple_snapshots (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:46:03.883ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:46:03.883ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:46:03.883ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:46:03.883ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:46:03.884ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:46:03.888ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:46:03.892ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:46:03.892ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:46:03.892ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:46:03.892ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:46:03.892ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:46:03.894ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:46:03.899ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:46:03.910ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:46:03.940ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:46:03.940ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:46:03.940ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:46:03.940ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:46:03.941ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:46:03.956ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:46:03.964ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:46:03.964ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:46:03.964ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:46:03.964ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:46:03.964ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:46:03.965ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:46:03.967ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:46:03.975ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:46:03.982ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:46:03.982ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:46:03.982ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:46:03.982ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:46:03.983ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:46:03.987ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:46:03.998ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:46:04.031ZINFOtest_multiple_snapshots (InternalApi): Created recovery silo
16382025-04-30T06:46:04.034ZINFOtest_multiple_snapshots (InternalApi): Created recovery user
16392025-04-30T06:46:04.039ZINFOtest_multiple_snapshots (InternalApi): Created recovery user's password
16402025-04-30T06:46:04.040ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, lookup_type: ById(0717693e-6ab1-4ee8-9f9f-b706e2906a23) }
16412025-04-30T06:46:04.040ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 0717693e-6ab1-4ee8-9f9f-b706e2906a23 resource_type = Silo
16422025-04-30T06:46:04.040ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:46:04.040ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:46:04.040ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:46:04.044ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:46:04.044ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:46:04.044ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:46:04.044ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:46:04.054ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:46:04.058ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, lookup_type: ById(0717693e-6ab1-4ee8-9f9f-b706e2906a23) } result = Ok(())
16512025-04-30T06:46:04.058ZDEBGtest_multiple_snapshots (InternalApi): Generated role assignment queries
16522025-04-30T06:46:04.062ZINFOtest_multiple_snapshots (InternalApi): Granted Silo privileges
16532025-04-30T06:46:04.062ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:46:04.062ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:46:04.062ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:46:04.062ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:46:04.063ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:46:04.066ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:46:04.070ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:46:04.088ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:46:04.088ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:46:04.089ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:46:04.094ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:46:04.094ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:46:04.095ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:46:04.100ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:46:04.100ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:46:04.102ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:46:04.110ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:46:04.110ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:46:04.110ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:46:04.110ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:46:04.110ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:46:04.117ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:46:04.117ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:46:04.117ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:46:04.117ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:46:04.124ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:46:04.124ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:46:04.124ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:46:04.125ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:46:04.131ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:46:04.141ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:46:04.141ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:46:04.141ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:46:04.141ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:46:04.141ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:46:04.141ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:46:04.142ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:46:04.145ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:46:04.145ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:46:04.145ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:46:04.147ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:46:04.148ZINFOtest_multiple_snapshots (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 51e79bd8-2b58-4bd1-af56-8cf2cb6962ea
16952025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 51e79bd8-2b58-4bd1-af56-8cf2cb6962ea time_started = 2025-04-30 06:46:03.005884 UTC
16962025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.037786108s iteration = 2 status = {"collection_id":"51e79bd8-2b58-4bd1-af56-8cf2cb6962ea","time_done":"2025-04-30 06:46:03.703222 UTC","time_started":"2025-04-30 06:46:03.005884 UTC"}
16972025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
16982025-04-30T06:46:04.148ZWARNtest_multiple_snapshots (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
16992025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 51.783\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17002025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17012025-04-30T06:46:04.148ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 11.303\u{b5}s iteration = 2 status = {"error":"task disabled"}
17022025-04-30T06:46:04.150ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17032025-04-30T06:46:04.150ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:46:04.150ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:46:04.151ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:46:04.154ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17072025-04-30T06:46:04.164ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17082025-04-30T06:46:04.172ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17092025-04-30T06:46:04.173ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17102025-04-30T06:46:04.173ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17112025-04-30T06:46:04.196ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17122025-04-30T06:46:04.196ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17132025-04-30T06:46:04.196ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17142025-04-30T06:46:04.201ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:46:04.201ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:46:04.202ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:46:04.205ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17182025-04-30T06:46:04.205ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17192025-04-30T06:46:04.206ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17202025-04-30T06:46:04.209ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17212025-04-30T06:46:04.210ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17222025-04-30T06:46:04.210ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:46:04.210ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:46:04.211ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:46:04.213ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17262025-04-30T06:46:04.214ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17272025-04-30T06:46:04.214ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17282025-04-30T06:46:04.214ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17292025-04-30T06:46:04.217ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17302025-04-30T06:46:04.217ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17312025-04-30T06:46:04.217ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17322025-04-30T06:46:04.218ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17332025-04-30T06:46:04.220ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17342025-04-30T06:46:04.229ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17352025-04-30T06:46:04.229ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17362025-04-30T06:46:04.229ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17372025-04-30T06:46:04.229ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17382025-04-30T06:46:04.230ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17392025-04-30T06:46:04.233ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17402025-04-30T06:46:04.233ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17412025-04-30T06:46:04.234ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17422025-04-30T06:46:04.238ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:46:04.238ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:46:04.239ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:46:04.251ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17462025-04-30T06:46:04.251ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17472025-04-30T06:46:04.254ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17482025-04-30T06:46:04.263ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17492025-04-30T06:46:04.264ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17502025-04-30T06:46:04.265ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:46:04.265ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:46:04.267ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:46:04.272ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17542025-04-30T06:46:04.272ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17552025-04-30T06:46:04.272ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17562025-04-30T06:46:04.273ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17572025-04-30T06:46:04.275ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17582025-04-30T06:46:04.275ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:46:04.275ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:46:04.276ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:46:04.278ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17622025-04-30T06:46:04.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
17632025-04-30T06:46:04.287ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17642025-04-30T06:46:04.287ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17652025-04-30T06:46:04.287ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17662025-04-30T06:46:04.287ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:46:04.287ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:46:04.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:46:04.290ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:46:04.290ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:46:04.290ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:46:04.291ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:46:04.293ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:46:04.293ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:46:04.293ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:46:04.294ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:46:04.304ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:46:04.312ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
17792025-04-30T06:46:04.312ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:46:04.312ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:46:04.312ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:46:04.317ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:46:04.317ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:46:04.318ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:46:04.323ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:46:04.323ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:46:04.324ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:46:04.327ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:46:04.327ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:46:04.331ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:46:04.335ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:46:04.335ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:46:04.337ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:46:04.344ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17952025-04-30T06:46:04.344ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:46:04.344ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:46:04.344ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:46:04.347ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:46:04.350ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:46:04.350ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:46:04.350ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:46:04.351ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:46:04.357ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:46:04.357ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:46:04.357ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:46:04.357ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:46:04.383ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:46:04.399ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
18092025-04-30T06:46:04.399ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:46:04.399ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
18112025-04-30T06:46:04.399ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:46:04.399ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:46:04.399ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:46:04.400ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:46:04.402ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:46:04.402ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:46:04.402ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:46:04.410ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:46:04.418ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:46:04.418ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:46:04.418ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:46:04.419ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:46:04.430ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:46:04.455ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
18252025-04-30T06:46:04.455ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:46:04.455ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:46:04.457ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:46:04.493ZDEBGtest_multiple_snapshots (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 0f055d25-99fb-4eaf-9d8c-68300c343520, kind: Service { id: 75069e0e-8b61-4a18-8a45-042e87d80ea0 }, name: Name("external-dns-75069e0e-8b61-4a18-8a45-042e87d80ea0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: c57b1224-989a-4249-bb7c-a123febe26f7, kind: Service { id: 3ae6adf1-675a-4c6a-a06e-473e2878df6e }, name: Name("nexus-3ae6adf1-675a-4c6a-a06e-473e2878df6e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:46:04.493ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:46:04.493ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:46:04.494ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:46:04.494ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:46:04.494ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:46:04.523ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:46:04.527ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:46:04.527ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:46:04.527ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:46:04.528ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:46:04.554ZTRACtest_multiple_snapshots (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:46:04.070482Z, allowed_ips: None }
18402025-04-30T06:46:04.554ZDEBGtest_multiple_snapshots (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:46:04.555ZDEBGtest_multiple_snapshots (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0f055d25-99fb-4eaf-9d8c-68300c343520, kind: Service { id: 75069e0e-8b61-4a18-8a45-042e87d80ea0 }, name: Name("external-dns-75069e0e-8b61-4a18-8a45-042e87d80ea0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: c57b1224-989a-4249-bb7c-a123febe26f7, kind: Service { id: 3ae6adf1-675a-4c6a-a06e-473e2878df6e }, name: Name("nexus-3ae6adf1-675a-4c6a-a06e-473e2878df6e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:46:04.555ZDEBGtest_multiple_snapshots (nexus): resolved 2 rules for sleds
18432025-04-30T06:46:04.675ZDEBGtest_multiple_snapshots (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:45:58.242451Z, time_modified: 2025-04-30T06:45:58.242451Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(54590), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(45115) }]
18442025-04-30T06:46:04.675ZDEBGtest_multiple_snapshots (nexus): sending firewall rules to sled agents
18452025-04-30T06:46:04.675ZTRACtest_multiple_snapshots (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:46:04.675ZDEBGtest_multiple_snapshots (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:46:04.676ZDEBGtest_multiple_snapshots (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:46:04.684ZTRACtest_multiple_snapshots (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:46:04.684ZTRACtest_multiple_snapshots (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:46:04.684ZTRACtest_multiple_snapshots (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:46:04.684ZDEBGtest_multiple_snapshots (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:46:04.685ZDEBGtest_multiple_snapshots (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:46:04.687ZDEBGtest_multiple_snapshots (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:46:04.691ZDEBGtest_multiple_snapshots (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:46:04.727ZDEBGtest_multiple_snapshots (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:54590/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:46:04.727ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:56412 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:46:04.728ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:56412 req_id = 3b79f9ca-cce8-4e8e-b7bd-ee2eb5ba22b5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:46:04.739ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 10940 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:56412 req_id = 3b79f9ca-cce8-4e8e-b7bd-ee2eb5ba22b5 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:46:04.739ZDEBGtest_multiple_snapshots (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:54590/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "3b79f9ca-cce8-4e8e-b7bd-ee2eb5ba22b5", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
18602025-04-30T06:46:04.741ZINFOtest_multiple_snapshots (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:46:04.741ZTRACtest_multiple_snapshots (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:46:04.741ZDEBGtest_multiple_snapshots (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:46:04.742ZDEBGtest_multiple_snapshots (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:46:04.742ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:46:04.742ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:46:04.742ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:46:04.742ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:46:04.743ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:46:04.743ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:46:04.743ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:46:04.744ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:46:04.744ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:46:04.744ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:46:04.744ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:46:04.745ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:46:04.745ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:46:04.745ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:46:04.748ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:46:04.748ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:46:04.748ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:46:04.748ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:46:04.748ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:46:04.748ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:46:04.749ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:46:04.749ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 4a 6b 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 Jk..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:65311 req_id = a4e7c2cc-3361-4d18-b8fc-172116d97c70
18912025-04-30T06:46:04.749ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19051,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65311 req_id = a4e7c2cc-3361-4d18-b8fc-172116d97c70
18922025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (store): query key key = _nameservice._tcp
18932025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (store): query key key = 876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host
18942025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65311 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35840, target: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal") })) }] req_id = a4e7c2cc-3361-4d18-b8fc-172116d97c70
18952025-04-30T06:46:04.750ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: b6 08 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59741 req_id = c50e1e44-b7f3-4d0f-819b-99f0a28de26e
18962025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46600,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59741 req_id = c50e1e44-b7f3-4d0f-819b-99f0a28de26e
18972025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (store): query key key = _external-dns._tcp
18982025-04-30T06:46:04.750ZDEBGtest_multiple_snapshots (store): query key key = 75069e0e-8b61-4a18-8a45-042e87d80ea0.host
18992025-04-30T06:46:04.751ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59741 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 42239, target: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal") })) }] req_id = c50e1e44-b7f3-4d0f-819b-99f0a28de26e
19002025-04-30T06:46:04.751ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35840, target: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4429, tv_nsec: 550397476 } })
19012025-04-30T06:46:04.751ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:46:04.751ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 42239, target: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4429, tv_nsec: 550675079 } })
19032025-04-30T06:46:04.751ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:46:04.752ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:46:04.752ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:46:04.752ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:46:04.752ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:46:04.752ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:46:04.752ZTRACtest_multiple_snapshots (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:46:04.753ZTRACtest_multiple_snapshots (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:46:04.753ZTRACtest_multiple_snapshots (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:46:04.753ZDEBGtest_multiple_snapshots (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:46:04.753ZDEBGtest_multiple_snapshots (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:46:04.753ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: a1 91 01 00 00 01 00 00 00 00 00 01 24 38 37 36 ............$876\n0010: 64 63 65 65 62 2d 62 36 64 34 2d 34 65 32 62 2d dceeb-b6d4-4e2b-\n0020: 62 35 30 62 2d 65 31 63 30 36 65 31 31 39 36 66 b50b-e1c06e1196f\n0030: 32 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 2.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:64736 req_id = 9f51f2ad-7980-47e6-b5d9-9da2ffa28646
19152025-04-30T06:46:04.753ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 56,\n 55,\n 54,\n 100,\n 99,\n 101,\n 101,\n 98,\n 45,\n 98,\n 54,\n 100,\n 52,\n 45,\n 52,\n 101,\n 50,\n 98,\n 45,\n 98,\n 53,\n 48,\n 98,\n 45,\n 101,\n 49,\n 99,\n 48,\n 54,\n 101,\n 49,\n 49,\n 57,\n 54,\n 102,\n 50,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64736 req_id = 9f51f2ad-7980-47e6-b5d9-9da2ffa28646
19162025-04-30T06:46:04.753ZDEBGtest_multiple_snapshots (store): query key key = 876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host
19172025-04-30T06:46:04.753ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64736 query = LowerQuery { name: LowerName(Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal.")), original: Query { name: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("876dceeb-b6d4-4e2b-b50b-e1c06e1196f2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9f51f2ad-7980-47e6-b5d9-9da2ffa28646
19182025-04-30T06:46:04.753ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: cb 35 01 00 00 01 00 00 00 00 00 01 24 37 35 30 .5..........$750\n0010: 36 39 65 30 65 2d 38 62 36 31 2d 34 61 31 38 2d 69e0e-8b61-4a18-\n0020: 38 61 34 35 2d 30 34 32 65 38 37 64 38 30 65 61 8a45-042e87d80ea\n0030: 30 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 0.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:56232 req_id = d361647e-25b2-44f0-bcff-21ce9419bd15
19192025-04-30T06:46:04.754ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52021,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 55,\n 53,\n 48,\n 54,\n 57,\n 101,\n 48,\n 101,\n 45,\n 56,\n 98,\n 54,\n 49,\n 45,\n 52,\n 97,\n 49,\n 56,\n 45,\n 56,\n 97,\n 52,\n 53,\n 45,\n 48,\n 52,\n 50,\n 101,\n 56,\n 55,\n 100,\n 56,\n 48,\n 101,\n 97,\n 48,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56232 req_id = d361647e-25b2-44f0-bcff-21ce9419bd15
19202025-04-30T06:46:04.754ZDEBGtest_multiple_snapshots (store): query key key = 75069e0e-8b61-4a18-8a45-042e87d80ea0.host
19212025-04-30T06:46:04.754ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56232 query = LowerQuery { name: LowerName(Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal.")), original: Query { name: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("75069e0e-8b61-4a18-8a45-042e87d80ea0.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d361647e-25b2-44f0-bcff-21ce9419bd15
19222025-04-30T06:46:04.754ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19232025-04-30T06:46:04.754ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19242025-04-30T06:46:04.754ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19252025-04-30T06:46:04.754ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19262025-04-30T06:46:04.754ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19272025-04-30T06:46:04.755ZINFOtest_multiple_snapshots (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:35840] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:46:04.755ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.756464ms iteration = 2 status = {"addresses":["[::1]:35840"]}
19292025-04-30T06:46:04.755ZINFOtest_multiple_snapshots (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:42239] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:46:04.755ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.238041ms iteration = 2 status = {"addresses":["[::1]:42239"]}
19312025-04-30T06:46:04.755ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:46:04.755ZWARNtest_multiple_snapshots (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:46:04.755ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 21.542\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:46:04.755ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:46:04.755ZWARNtest_multiple_snapshots (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:46:04.756ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 11.233\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:46:04.763ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:46:04.767ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:46:04.767ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:46:04.767ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:46:04.768ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:46:04.768ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:46:04.771ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:46:04.771ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:46:04.771ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:46:04.772ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:46:04.772ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:46:04.777ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:46:04.779ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-04-30T06:46:04.787ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19512025-04-30T06:46:04.790ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19522025-04-30T06:46:04.790ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19532025-04-30T06:46:04.790ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19542025-04-30T06:46:04.790ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19552025-04-30T06:46:04.790ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19562025-04-30T06:46:04.790ZDEBGtest_multiple_snapshots (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19572025-04-30T06:46:04.799ZDEBGtest_multiple_snapshots (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19582025-04-30T06:46:04.800ZINFOtest_multiple_snapshots (nexus): Rack initialized
19592025-04-30T06:46:04.800ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19602025-04-30T06:46:04.800ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
19612025-04-30T06:46:04.801ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19622025-04-30T06:46:04.802ZDEBGtest_multiple_snapshots (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19632025-04-30T06:46:04.803ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19642025-04-30T06:46:04.803ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19652025-04-30T06:46:04.803ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19662025-04-30T06:46:04.803ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19672025-04-30T06:46:04.803ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19682025-04-30T06:46:04.803ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19692025-04-30T06:46:04.803ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19702025-04-30T06:46:04.803ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
19712025-04-30T06:46:04.803ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19722025-04-30T06:46:04.804ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-04-30T06:46:04.812ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19742025-04-30T06:46:04.812ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19752025-04-30T06:46:04.812ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19762025-04-30T06:46:04.813ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19772025-04-30T06:46:04.816ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19782025-04-30T06:46:04.824ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19792025-04-30T06:46:04.824ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19802025-04-30T06:46:04.824ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19812025-04-30T06:46:04.828ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19822025-04-30T06:46:04.829ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19832025-04-30T06:46:04.831ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
19842025-04-30T06:46:04.831ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19852025-04-30T06:46:04.831ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c3 ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57973 req_id = 07c79e5c-8ed1-4d98-a5fd-382ab8a77c34
19862025-04-30T06:46:04.831ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50091,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57973 req_id = 07c79e5c-8ed1-4d98-a5fd-382ab8a77c34
19872025-04-30T06:46:04.831ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
19882025-04-30T06:46:04.833ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19892025-04-30T06:46:04.833ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57973 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 07c79e5c-8ed1-4d98-a5fd-382ab8a77c34
19902025-04-30T06:46:04.834ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4429, tv_nsec: 633523690 } })
19912025-04-30T06:46:04.834ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
19922025-04-30T06:46:04.835ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 63 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e c...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51732 req_id = aee521bc-fd10-41b3-865a-3f4c724ffb50
19932025-04-30T06:46:04.835ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25375,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51732 req_id = aee521bc-fd10-41b3-865a-3f4c724ffb50
19942025-04-30T06:46:04.835ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19952025-04-30T06:46:04.835ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51732 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = aee521bc-fd10-41b3-865a-3f4c724ffb50
19962025-04-30T06:46:04.835ZDEBGtest_multiple_snapshots (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
19972025-04-30T06:46:04.836ZDEBGtest_multiple_snapshots (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c36909c7-8d72-4246-a1bd-71c10e179b01 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
19982025-04-30T06:46:04.836ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19992025-04-30T06:46:04.836ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20002025-04-30T06:46:04.836ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20012025-04-30T06:46:04.836ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20022025-04-30T06:46:04.837ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20032025-04-30T06:46:04.839ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20042025-04-30T06:46:04.839ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
20052025-04-30T06:46:04.840ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20062025-04-30T06:46:04.944ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20072025-04-30T06:46:04.947ZDEBGtest_multiple_snapshots (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
20082025-04-30T06:46:04.947ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
20092025-04-30T06:46:04.947ZDEBGtest_multiple_snapshots (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
20102025-04-30T06:46:04.947ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20112025-04-30T06:46:04.947ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20122025-04-30T06:46:04.947ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20132025-04-30T06:46:04.947ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20142025-04-30T06:46:04.948ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20152025-04-30T06:46:04.948ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20162025-04-30T06:46:04.948ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20172025-04-30T06:46:04.948ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20182025-04-30T06:46:04.948ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20192025-04-30T06:46:04.950ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20202025-04-30T06:46:04.950ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20212025-04-30T06:46:04.957ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20222025-04-30T06:46:04.957ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20232025-04-30T06:46:04.957ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20242025-04-30T06:46:04.960ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20252025-04-30T06:46:04.962ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b2 93 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55824 req_id = 7b7d59b3-162d-44ce-8548-5bb2ba7e9563
20262025-04-30T06:46:04.962ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45715,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55824 req_id = 7b7d59b3-162d-44ce-8548-5bb2ba7e9563
20272025-04-30T06:46:04.962ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
20282025-04-30T06:46:04.962ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45715, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55824 req_id = 7b7d59b3-162d-44ce-8548-5bb2ba7e9563
20292025-04-30T06:46:04.962ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20302025-04-30T06:46:04.970ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20312025-04-30T06:46:04.970ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20322025-04-30T06:46:04.970ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20332025-04-30T06:46:04.970ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20342025-04-30T06:46:04.971ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20352025-04-30T06:46:04.975ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20362025-04-30T06:46:04.976ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20372025-04-30T06:46:04.976ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20382025-04-30T06:46:04.976ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20392025-04-30T06:46:04.977ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20402025-04-30T06:46:04.977ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20412025-04-30T06:46:04.977ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20422025-04-30T06:46:04.977ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
20432025-04-30T06:46:04.978ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20442025-04-30T06:46:04.978ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
20452025-04-30T06:46:04.978ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
20462025-04-30T06:46:04.978ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
20472025-04-30T06:46:04.979ZDEBGtest_multiple_snapshots (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c36909c7-8d72-4246-a1bd-71c10e179b01 dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20482025-04-30T06:46:04.979ZDEBGtest_multiple_snapshots (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20492025-04-30T06:46:04.980ZINFOtest_multiple_snapshots (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20502025-04-30T06:46:04.980ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 236.299531ms iteration = 2 status = {"generation":2}
20512025-04-30T06:46:04.980ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20522025-04-30T06:46:05.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:42239 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:42239] } uri = http://[::1]:42239/config
20532025-04-30T06:46:05.048ZDEBGtest_multiple_snapshots (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20542025-04-30T06:46:05.048ZDEBGtest_multiple_snapshots (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 53fbdece-05ed-4ae5-bb5c-271fc96838a2 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20552025-04-30T06:46:05.049ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20562025-04-30T06:46:05.049ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20572025-04-30T06:46:05.049ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20582025-04-30T06:46:05.049ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20592025-04-30T06:46:05.050ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20602025-04-30T06:46:05.052ZINFOtest_multiple_snapshots (http): accepted connection kind = dns local_addr = [::1]:42239 remote_addr = [::1]:57875
20612025-04-30T06:46:05.060ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20622025-04-30T06:46:05.060ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20632025-04-30T06:46:05.060ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
20642025-04-30T06:46:05.061ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20652025-04-30T06:46:05.062ZTRACtest_multiple_snapshots (http): incoming request kind = dns local_addr = [::1]:42239 method = PUT remote_addr = [::1]:57875 req_id = 9a04a4f8-d035-455c-a7b5-c76326c793d4 uri = /config
20662025-04-30T06:46:05.062ZINFOtest_multiple_snapshots (store): attempting generation update new_generation = 2 req_id = 9a04a4f8-d035-455c-a7b5-c76326c793d4
20672025-04-30T06:46:05.062ZINFOtest_multiple_snapshots (store): pruning trees for generations newer than 0
20682025-04-30T06:46:05.062ZDEBGtest_multiple_snapshots (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20692025-04-30T06:46:05.063ZDEBGtest_multiple_snapshots (store): updating current config
20702025-04-30T06:46:05.063ZDEBGtest_multiple_snapshots (store): flushing default tree
20712025-04-30T06:46:05.063ZINFOtest_multiple_snapshots (store): pruning trees for generations older than 2 keep = 3
20722025-04-30T06:46:05.063ZINFOtest_multiple_snapshots (store): updated generation new_generation = 2 req_id = 9a04a4f8-d035-455c-a7b5-c76326c793d4
20732025-04-30T06:46:05.063ZINFOtest_multiple_snapshots (http): request completed kind = dns latency_us = 1414 local_addr = [::1]:42239 method = PUT remote_addr = [::1]:57875 req_id = 9a04a4f8-d035-455c-a7b5-c76326c793d4 response_code = 204 uri = /config
20742025-04-30T06:46:05.064ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:42239 generation = 2 result = Ok(Response { url: "http://[::1]:42239/config", status: 204, headers: {"x-request-id": "9a04a4f8-d035-455c-a7b5-c76326c793d4", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} }) servers = DnsServersList { addresses: [[::1]:42239] }
20752025-04-30T06:46:05.064ZINFOtest_multiple_snapshots (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:42239 generation = 2 servers = DnsServersList { addresses: [[::1]:42239] }
20762025-04-30T06:46:05.069ZINFOtest_multiple_snapshots (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:42239] }
20772025-04-30T06:46:05.069ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 89.084585ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:42239":{"Ok":null}}}
20782025-04-30T06:46:05.069ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20792025-04-30T06:46:05.077ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20802025-04-30T06:46:05.077ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20812025-04-30T06:46:05.077ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20822025-04-30T06:46:05.077ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20832025-04-30T06:46:05.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a079ed62-ab74-40a9-9705-7ad49c4aef32", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
20842025-04-30T06:46:05.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
20852025-04-30T06:46:05.082ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20862025-04-30T06:46:05.082ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20872025-04-30T06:46:05.082ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
20882025-04-30T06:46:05.084ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20892025-04-30T06:46:05.084ZDEBGtest_multiple_snapshots (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 53fbdece-05ed-4ae5-bb5c-271fc96838a2 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20902025-04-30T06:46:05.084ZDEBGtest_multiple_snapshots (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20912025-04-30T06:46:05.084ZINFOtest_multiple_snapshots (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20922025-04-30T06:46:05.084ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 342.29488ms iteration = 2 status = {"generation":1}
20932025-04-30T06:46:05.084ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20942025-04-30T06:46:05.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:35840 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:35840] } uri = http://[::1]:35840/config
20952025-04-30T06:46:05.184ZINFOtest_multiple_snapshots (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:45:56.716501Z, time_modified: 2025-04-30T06:45:56.716501Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, db_silo: Silo { identity: SiloIdentity { id: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:46:03.883514Z, time_modified: 2025-04-30T06:46:03.883514Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, db_silo: Silo { identity: SiloIdentity { id: 0717693e-6ab1-4ee8-9f9f-b706e2906a23, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:46:03.883514Z, time_modified: 2025-04-30T06:46:03.883514Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20962025-04-30T06:46:05.184ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 439.798239ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"0717693e-6ab1-4ee8-9f9f-b706e2906a23","tls_certs":[]}},"default_endpoint":{"silo_id":"0717693e-6ab1-4ee8-9f9f-b706e2906a23","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 0717693e-6ab1-4ee8-9f9f-b706e2906a23 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20972025-04-30T06:46:05.185ZINFOtest_multiple_snapshots (http): accepted connection kind = dns local_addr = [::1]:35840 remote_addr = [::1]:41452
20982025-04-30T06:46:05.186ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20992025-04-30T06:46:05.194ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21002025-04-30T06:46:05.194ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21012025-04-30T06:46:05.194ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21022025-04-30T06:46:05.194ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21032025-04-30T06:46:05.194ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21042025-04-30T06:46:05.194ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21052025-04-30T06:46:05.194ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21062025-04-30T06:46:05.194ZTRACtest_multiple_snapshots (http): incoming request kind = dns local_addr = [::1]:35840 method = PUT remote_addr = [::1]:41452 req_id = 56a667d8-cf2a-43fe-a181-ed677b6d2793 uri = /config
21072025-04-30T06:46:05.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5bc4929-cf1d-43b4-ae30-24b19b53045c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21082025-04-30T06:46:05.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
21092025-04-30T06:46:05.195ZINFOtest_multiple_snapshots (store): attempting generation update new_generation = 1 req_id = 56a667d8-cf2a-43fe-a181-ed677b6d2793
21102025-04-30T06:46:05.195ZINFOtest_multiple_snapshots (store): updated generation new_generation = 1 req_id = 56a667d8-cf2a-43fe-a181-ed677b6d2793
21112025-04-30T06:46:05.196ZINFOtest_multiple_snapshots (http): request completed kind = dns latency_us = 1271 local_addr = [::1]:35840 method = PUT remote_addr = [::1]:41452 req_id = 56a667d8-cf2a-43fe-a181-ed677b6d2793 response_code = 204 uri = /config
21122025-04-30T06:46:05.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:35840 generation = 1 result = Ok(Response { url: "http://[::1]:35840/config", status: 204, headers: {"x-request-id": "56a667d8-cf2a-43fe-a181-ed677b6d2793", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} }) servers = DnsServersList { addresses: [[::1]:35840] }
21132025-04-30T06:46:05.197ZINFOtest_multiple_snapshots (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:35840 generation = 1 servers = DnsServersList { addresses: [[::1]:35840] }
21142025-04-30T06:46:05.198ZINFOtest_multiple_snapshots (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:35840] }
21152025-04-30T06:46:05.198ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 113.935148ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:35840":{"Ok":null}}}
21162025-04-30T06:46:05.199ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21172025-04-30T06:46:05.199ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:46:05.199ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:46:05.199ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:46:05.202ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21212025-04-30T06:46:05.202ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21222025-04-30T06:46:05.202ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21232025-04-30T06:46:05.203ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21242025-04-30T06:46:05.204ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9281694-0cad-4444-b477-3432f6c4ea3a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21252025-04-30T06:46:05.204ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
21262025-04-30T06:46:05.208ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21272025-04-30T06:46:05.217ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21282025-04-30T06:46:05.217ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21292025-04-30T06:46:05.217ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21302025-04-30T06:46:05.218ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21312025-04-30T06:46:05.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02e45bfb-9cbd-4da9-a68b-29c629f97469", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21322025-04-30T06:46:05.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
21332025-04-30T06:46:05.221ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21342025-04-30T06:46:05.221ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21352025-04-30T06:46:05.222ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21362025-04-30T06:46:05.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b81b066-3312-4e3c-820b-5a8eae4211e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21372025-04-30T06:46:05.233ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
21382025-04-30T06:46:05.236ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:46:05.236ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:46:05.238ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:46:05.243ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21422025-04-30T06:46:05.243ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21432025-04-30T06:46:05.244ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21442025-04-30T06:46:05.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ca43a2f-58f5-4c0c-a9b0-44c356fc306a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21452025-04-30T06:46:05.246ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
21462025-04-30T06:46:05.248ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21472025-04-30T06:46:05.248ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21482025-04-30T06:46:05.248ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21492025-04-30T06:46:05.248ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21502025-04-30T06:46:05.249ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21512025-04-30T06:46:05.253ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21522025-04-30T06:46:05.253ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21532025-04-30T06:46:05.253ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21542025-04-30T06:46:05.254ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21552025-04-30T06:46:05.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3457f40a-43a3-4a91-a5b5-ec2ba16f9968", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21562025-04-30T06:46:05.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
21572025-04-30T06:46:05.259ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21582025-04-30T06:46:05.259ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21592025-04-30T06:46:05.259ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21602025-04-30T06:46:05.260ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21612025-04-30T06:46:05.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "191346e7-8a3d-4932-a65d-9c47f0cc7328", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21622025-04-30T06:46:05.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
21632025-04-30T06:46:05.266ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "115208c9-917b-40ef-a813-2a71339fe43d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21642025-04-30T06:46:05.266ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
21652025-04-30T06:46:05.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46eb9412-135f-46f4-9494-848ba7b68440", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21662025-04-30T06:46:05.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
21672025-04-30T06:46:05.269ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21682025-04-30T06:46:05.278ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21692025-04-30T06:46:05.278ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21702025-04-30T06:46:05.279ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21712025-04-30T06:46:05.279ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21722025-04-30T06:46:05.281ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21732025-04-30T06:46:05.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d44a4df-a974-43c1-832b-273f4907e38f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21742025-04-30T06:46:05.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
21752025-04-30T06:46:05.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ed4748-4bed-4294-8265-7d6f1b1e9dc3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21762025-04-30T06:46:05.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
21772025-04-30T06:46:05.286ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21782025-04-30T06:46:05.286ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21792025-04-30T06:46:05.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21802025-04-30T06:46:05.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df67bf1a-d264-403b-ae12-1b9ffbed7f3c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21812025-04-30T06:46:05.289ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
21822025-04-30T06:46:05.293ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21832025-04-30T06:46:05.293ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21842025-04-30T06:46:05.293ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21852025-04-30T06:46:05.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea6352b-c931-4787-9168-8cee5eec2ac2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21862025-04-30T06:46:05.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
21872025-04-30T06:46:05.296ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21882025-04-30T06:46:05.296ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21892025-04-30T06:46:05.296ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21902025-04-30T06:46:05.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4522f56b-8e51-4f58-8989-7437bad6c0b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21912025-04-30T06:46:05.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
21922025-04-30T06:46:05.300ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21932025-04-30T06:46:05.300ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21942025-04-30T06:46:05.300ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21952025-04-30T06:46:05.300ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
21962025-04-30T06:46:05.300ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21972025-04-30T06:46:05.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "121c793f-bf70-4e2d-9fc6-a5f727e55bda", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
21982025-04-30T06:46:05.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
21992025-04-30T06:46:05.303ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22002025-04-30T06:46:05.303ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22012025-04-30T06:46:05.303ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22022025-04-30T06:46:05.304ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22032025-04-30T06:46:05.308ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22042025-04-30T06:46:05.308ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22052025-04-30T06:46:05.308ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22062025-04-30T06:46:05.309ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22072025-04-30T06:46:05.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd09c2a6-6062-4368-98bc-f28372587b71", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22082025-04-30T06:46:05.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
22092025-04-30T06:46:05.314ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22102025-04-30T06:46:05.324ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22112025-04-30T06:46:05.324ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22122025-04-30T06:46:05.324ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22132025-04-30T06:46:05.324ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22142025-04-30T06:46:05.324ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22152025-04-30T06:46:05.324ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22162025-04-30T06:46:05.325ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22172025-04-30T06:46:05.331ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22182025-04-30T06:46:05.331ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22192025-04-30T06:46:05.331ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22202025-04-30T06:46:05.332ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22212025-04-30T06:46:05.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c96f36-88bc-45a7-88cf-690c111a878a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22222025-04-30T06:46:05.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
22232025-04-30T06:46:05.336ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22242025-04-30T06:46:05.336ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22252025-04-30T06:46:05.336ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22262025-04-30T06:46:05.337ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22272025-04-30T06:46:05.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0de0b1a2-76e4-4aa9-97fe-2bc800e88c10", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22282025-04-30T06:46:05.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
22292025-04-30T06:46:05.339ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22302025-04-30T06:46:05.345ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22312025-04-30T06:46:05.345ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22322025-04-30T06:46:05.345ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22332025-04-30T06:46:05.346ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22342025-04-30T06:46:05.346ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7fa9766-bed3-471c-a66b-9c15f249509e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22352025-04-30T06:46:05.346ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
22362025-04-30T06:46:05.347ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d02abbf-1ca5-4e73-ba01-a170cfe45b21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22372025-04-30T06:46:05.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
22382025-04-30T06:46:05.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "227ca2ed-d93c-4607-8ee7-e7b5378b2d60", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22392025-04-30T06:46:05.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
22402025-04-30T06:46:05.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ebcc45-539a-45f3-8b84-88eac09eaf06", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22412025-04-30T06:46:05.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
22422025-04-30T06:46:05.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c6db32d-fa30-43be-a9da-82ebe00348f4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22432025-04-30T06:46:05.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
22442025-04-30T06:46:05.355ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22452025-04-30T06:46:05.355ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22462025-04-30T06:46:05.356ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22472025-04-30T06:46:05.358ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d282a07a-e9a3-41d5-b7ff-57ac38623699", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22482025-04-30T06:46:05.358ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
22492025-04-30T06:46:05.360ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22502025-04-30T06:46:05.360ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22512025-04-30T06:46:05.360ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22522025-04-30T06:46:05.362ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abfd84e8-10ed-44aa-b9a1-31c026c2f914", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22532025-04-30T06:46:05.362ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
22542025-04-30T06:46:05.368ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22552025-04-30T06:46:05.368ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22562025-04-30T06:46:05.369ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22572025-04-30T06:46:05.369ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edd14a6f-4c28-4b7c-bb09-0c5f23f0a8ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22582025-04-30T06:46:05.369ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
22592025-04-30T06:46:05.371ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b37e0ff4-1dc7-4904-a5bc-a3c8fc974d52", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22602025-04-30T06:46:05.371ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
22612025-04-30T06:46:05.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d8bfe05-c848-4d81-8c2c-ed7aa84b5c0e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22622025-04-30T06:46:05.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
22632025-04-30T06:46:05.376ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22642025-04-30T06:46:05.376ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22652025-04-30T06:46:05.376ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22662025-04-30T06:46:05.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2abf7365-69b3-4d3d-80ef-ec4d2b894f2f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22672025-04-30T06:46:05.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
22682025-04-30T06:46:05.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84146617-4647-477c-ae43-51e4d5f90d0c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22692025-04-30T06:46:05.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
22702025-04-30T06:46:05.379ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22712025-04-30T06:46:05.379ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22722025-04-30T06:46:05.379ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22732025-04-30T06:46:05.379ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22742025-04-30T06:46:05.380ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22752025-04-30T06:46:05.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce22dd7-1fec-44c4-83ae-3a1d59aa9bde", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22762025-04-30T06:46:05.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
22772025-04-30T06:46:05.383ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22782025-04-30T06:46:05.383ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22792025-04-30T06:46:05.383ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22802025-04-30T06:46:05.384ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22812025-04-30T06:46:05.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7fd1460-0ea8-47b7-88d7-8e547acb0911", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22822025-04-30T06:46:05.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
22832025-04-30T06:46:05.386ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22842025-04-30T06:46:05.386ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22852025-04-30T06:46:05.386ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22862025-04-30T06:46:05.387ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22872025-04-30T06:46:05.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf7a7d95-653c-4f44-a378-9b61f26ddd9f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22882025-04-30T06:46:05.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
22892025-04-30T06:46:05.390ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22902025-04-30T06:46:05.402ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22912025-04-30T06:46:05.402ZTRACtest_multiple_snapshots (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22922025-04-30T06:46:05.402ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22932025-04-30T06:46:05.402ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22942025-04-30T06:46:05.402ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22952025-04-30T06:46:05.402ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
22962025-04-30T06:46:05.403ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22972025-04-30T06:46:05.405ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90335c46-c29d-4dd6-8e48-e729dee31fad", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
22982025-04-30T06:46:05.405ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
22992025-04-30T06:46:05.405ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
23002025-04-30T06:46:05.405ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23012025-04-30T06:46:05.405ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23022025-04-30T06:46:05.413ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23032025-04-30T06:46:05.422ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23042025-04-30T06:46:05.422ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23052025-04-30T06:46:05.422ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23062025-04-30T06:46:05.423ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23072025-04-30T06:46:05.426ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23082025-04-30T06:46:05.434ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
23092025-04-30T06:46:05.434ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23102025-04-30T06:46:05.434ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23112025-04-30T06:46:05.435ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23122025-04-30T06:46:05.465ZDEBGtest_multiple_snapshots (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: c57b1224-989a-4249-bb7c-a123febe26f7, kind: Service { id: 3ae6adf1-675a-4c6a-a06e-473e2878df6e }, name: Name("nexus-3ae6adf1-675a-4c6a-a06e-473e2878df6e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 0f055d25-99fb-4eaf-9d8c-68300c343520, kind: Service { id: 75069e0e-8b61-4a18-8a45-042e87d80ea0 }, name: Name("external-dns-75069e0e-8b61-4a18-8a45-042e87d80ea0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
23132025-04-30T06:46:05.465ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23142025-04-30T06:46:05.465ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23152025-04-30T06:46:05.465ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23162025-04-30T06:46:05.465ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23172025-04-30T06:46:05.471ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23182025-04-30T06:46:05.487ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23192025-04-30T06:46:05.495ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23202025-04-30T06:46:05.495ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23212025-04-30T06:46:05.495ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23222025-04-30T06:46:05.496ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23232025-04-30T06:46:05.499ZTRACtest_multiple_snapshots (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:46:04.070482Z, allowed_ips: None }
23242025-04-30T06:46:05.500ZDEBGtest_multiple_snapshots (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23252025-04-30T06:46:05.501ZDEBGtest_multiple_snapshots (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0f055d25-99fb-4eaf-9d8c-68300c343520, kind: Service { id: 75069e0e-8b61-4a18-8a45-042e87d80ea0 }, name: Name("external-dns-75069e0e-8b61-4a18-8a45-042e87d80ea0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: c57b1224-989a-4249-bb7c-a123febe26f7, kind: Service { id: 3ae6adf1-675a-4c6a-a06e-473e2878df6e }, name: Name("nexus-3ae6adf1-675a-4c6a-a06e-473e2878df6e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23262025-04-30T06:46:05.501ZDEBGtest_multiple_snapshots (InternalApi): resolved 2 rules for sleds
23272025-04-30T06:46:05.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a9b90b-e42e-490c-b1ac-24588b11052e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23282025-04-30T06:46:05.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:46:05.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c125656-155c-46e0-bc5d-57ed2886c9a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23302025-04-30T06:46:05.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:46:05.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b39005b-7844-420a-b193-2555af43449f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23322025-04-30T06:46:05.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:46:05.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a600ed-55b4-46fa-8782-213e61a31585", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23342025-04-30T06:46:05.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
23352025-04-30T06:46:05.527ZDEBGtest_multiple_snapshots (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:45:58.242451Z, time_modified: 2025-04-30T06:45:58.242451Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(54590), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(45115) }]
23362025-04-30T06:46:05.528ZDEBGtest_multiple_snapshots (InternalApi): sending firewall rules to sled agents
23372025-04-30T06:46:05.528ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23382025-04-30T06:46:05.528ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23392025-04-30T06:46:05.534ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23402025-04-30T06:46:05.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ef32c85-c0c8-49a0-a8ba-de6c032e37b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23412025-04-30T06:46:05.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
23422025-04-30T06:46:05.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16c238ce-7b20-45fa-a151-007d797111c0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23432025-04-30T06:46:05.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
23442025-04-30T06:46:05.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09bb16c9-a137-43db-a13f-ff418c6189ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23452025-04-30T06:46:05.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
23462025-04-30T06:46:05.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25530cce-7846-4995-87d0-364762b430f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23472025-04-30T06:46:05.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
23482025-04-30T06:46:05.545ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23492025-04-30T06:46:05.545ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23502025-04-30T06:46:05.545ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23512025-04-30T06:46:05.545ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
23522025-04-30T06:46:05.546ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23532025-04-30T06:46:05.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9144b18f-be68-499c-8508-386124f431c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23542025-04-30T06:46:05.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
23552025-04-30T06:46:05.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62c8d1f9-d4a6-44e7-9fa7-ba7a7bfd381c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23562025-04-30T06:46:05.547ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23572025-04-30T06:46:05.547ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23582025-04-30T06:46:05.547ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23592025-04-30T06:46:05.547ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23602025-04-30T06:46:05.552ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23612025-04-30T06:46:05.558ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23622025-04-30T06:46:05.570ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23632025-04-30T06:46:05.646ZDEBGtest_multiple_snapshots (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:54590/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23642025-04-30T06:46:05.647ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23652025-04-30T06:46:05.654ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23662025-04-30T06:46:05.654ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23672025-04-30T06:46:05.654ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23682025-04-30T06:46:05.654ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23692025-04-30T06:46:05.654ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23702025-04-30T06:46:05.655ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23712025-04-30T06:46:05.658ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:61740 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23722025-04-30T06:46:05.659ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:61740 req_id = 7ff3bd65-af1f-4069-baa0-d6559dfd75d0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23732025-04-30T06:46:05.660ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 965 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:61740 req_id = 7ff3bd65-af1f-4069-baa0-d6559dfd75d0 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23742025-04-30T06:46:05.660ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23752025-04-30T06:46:05.669ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23762025-04-30T06:46:05.669ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23772025-04-30T06:46:05.669ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23782025-04-30T06:46:05.669ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23792025-04-30T06:46:05.670ZDEBGtest_multiple_snapshots (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:54590/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "7ff3bd65-af1f-4069-baa0-d6559dfd75d0", "date": "Wed, 30 Apr 2025 06:46:04 GMT"} })
23802025-04-30T06:46:05.672ZINFOtest_multiple_snapshots (InternalApi): updated firewall rules on 1 sleds
23812025-04-30T06:46:05.672ZINFOtest_multiple_snapshots (nexus): plumbed initial IP allowlist
23822025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = / versions = all
23832025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /access versions = all
23842025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /assets/{path} versions = all
23852025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /device/auth versions = all
23862025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /device/confirm versions = all
23872025-04-30T06:46:05.808ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /device/success versions = all
23882025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /device/token versions = all
23892025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /device/verify versions = all
23902025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/probes versions = all
23912025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /experimental/v1/probes versions = all
23922025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23932025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/probes/{probe} versions = all
23942025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/system/support-bundles versions = all
23952025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /experimental/v1/system/support-bundles versions = all
23962025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23972025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23982025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23992025-04-30T06:46:05.809ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24002025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24012025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24022025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
24032025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /images versions = all
24042025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /login versions = all
24052025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /login/{silo_name}/local versions = all
24062025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
24072025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
24082025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
24092025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /lookup/{path} versions = all
24102025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /projects/{path} versions = all
24112025-04-30T06:46:05.810ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /projects-new versions = all
24122025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /settings/{path} versions = all
24132025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /system/{path} versions = all
24142025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /utilization versions = all
24152025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/affinity-groups versions = all
24162025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/affinity-groups versions = all
24172025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
24182025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
24192025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24202025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24212025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24222025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24232025-04-30T06:46:05.811ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24242025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/anti-affinity-groups versions = all
24252025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/anti-affinity-groups versions = all
24262025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24272025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24282025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24292025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24302025-04-30T06:46:05.812ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24312025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24322025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24332025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/certificates versions = all
24342025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/certificates versions = all
24352025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/certificates/{certificate} versions = all
24362025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/certificates/{certificate} versions = all
24372025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/disks versions = all
24382025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/disks versions = all
24392025-04-30T06:46:05.813ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/disks/{disk} versions = all
24402025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/disks/{disk} versions = all
24412025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24422025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24432025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24442025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/disks/{disk}/finalize versions = all
24452025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24462025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/floating-ips versions = all
24472025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/floating-ips versions = all
24482025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24492025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24502025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24512025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24522025-04-30T06:46:05.814ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24532025-04-30T06:46:05.815ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/groups versions = all
24542025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/groups/{group_id} versions = all
24552025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/images versions = all
24562025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/images versions = all
24572025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/images/{image} versions = all
24582025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/images/{image} versions = all
24592025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/images/{image}/demote versions = all
24602025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/images/{image}/promote versions = all
24612025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances versions = all
24622025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances versions = all
24632025-04-30T06:46:05.816ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/instances/{instance} versions = all
24642025-04-30T06:46:05.817ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance} versions = all
24652025-04-30T06:46:05.817ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/instances/{instance} versions = all
24662025-04-30T06:46:05.817ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24672025-04-30T06:46:05.818ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24682025-04-30T06:46:05.818ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/disks versions = all
24692025-04-30T06:46:05.818ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24702025-04-30T06:46:05.818ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24712025-04-30T06:46:05.819ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/external-ips versions = all
24722025-04-30T06:46:05.819ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24732025-04-30T06:46:05.819ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24742025-04-30T06:46:05.820ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/reboot versions = all
24752025-04-30T06:46:05.820ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/serial-console versions = all
24762025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24772025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24782025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/start versions = all
24792025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/instances/{instance}/stop versions = all
24802025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24812025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24822025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24832025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/internet-gateway-ip-pools versions = all
24842025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/internet-gateway-ip-pools versions = all
24852025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24862025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/internet-gateways versions = all
24872025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/internet-gateways versions = all
24882025-04-30T06:46:05.821ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24892025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/internet-gateways/{gateway} versions = all
24902025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/ip-pools versions = all
24912025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/ip-pools/{pool} versions = all
24922025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/login/{silo_name}/local versions = all
24932025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/logout versions = all
24942025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/me versions = all
24952025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/me/groups versions = all
24962025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/me/ssh-keys versions = all
24972025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/me/ssh-keys versions = all
24982025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24992025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
25002025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/metrics/{metric_name} versions = all
25012025-04-30T06:46:05.822ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/network-interfaces versions = all
25022025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/network-interfaces versions = all
25032025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/network-interfaces/{interface} versions = all
25042025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/network-interfaces/{interface} versions = all
25052025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/network-interfaces/{interface} versions = all
25062025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/ping versions = all
25072025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/policy versions = all
25082025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/policy versions = all
25092025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/projects versions = all
25102025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/projects versions = all
25112025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/projects/{project} versions = all
25122025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/projects/{project} versions = all
25132025-04-30T06:46:05.823ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/projects/{project} versions = all
25142025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/projects/{project}/policy versions = all
25152025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/projects/{project}/policy versions = all
25162025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/snapshots versions = all
25172025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/snapshots versions = all
25182025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/snapshots/{snapshot} versions = all
25192025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/snapshots/{snapshot} versions = all
25202025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/disks versions = all
25212025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25222025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25232025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/racks versions = all
25242025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25252025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/sleds versions = all
25262025-04-30T06:46:05.824ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/hardware/sleds versions = all
25272025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25282025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25292025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25302025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25312025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25322025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/switch-port versions = all
25332025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25342025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25352025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25362025-04-30T06:46:05.825ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25372025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25382025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/switches versions = all
25392025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25402025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/identity-providers versions = all
25412025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/identity-providers/local/users versions = all
25422025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25432025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25442025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/identity-providers/saml versions = all
25452025-04-30T06:46:05.826ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25462025-04-30T06:46:05.827ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools versions = all
25472025-04-30T06:46:05.827ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools versions = all
25482025-04-30T06:46:05.827ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25492025-04-30T06:46:05.828ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools/{pool} versions = all
25502025-04-30T06:46:05.828ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25512025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25522025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25532025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25542025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25552025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25562025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25572025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25582025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25592025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools-service versions = all
25602025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25612025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25622025-04-30T06:46:05.829ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25632025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/metrics/{metric_name} versions = all
25642025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/address-lot versions = all
25652025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/address-lot versions = all
25662025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25672025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25682025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/allow-list versions = all
25692025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/networking/allow-list versions = all
25702025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/bfd-disable versions = all
25712025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/bfd-enable versions = all
25722025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bfd-status versions = all
25732025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/networking/bgp versions = all
25742025-04-30T06:46:05.830ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp versions = all
25752025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/bgp versions = all
25762025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25772025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25782025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25792025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25802025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-exported versions = all
25812025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-message-history versions = all
25822025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25832025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/bgp-status versions = all
25842025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/loopback-address versions = all
25852025-04-30T06:46:05.831ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/loopback-address versions = all
25862025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25872025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25882025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/switch-port-settings versions = all
25892025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/networking/switch-port-settings versions = all
25902025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25912025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/policy versions = all
25922025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/policy versions = all
25932025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/roles versions = all
25942025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/roles/{role_name} versions = all
25952025-04-30T06:46:05.832ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silo-quotas versions = all
25962025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silos versions = all
25972025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/silos versions = all
25982025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/system/silos/{silo} versions = all
25992025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silos/{silo} versions = all
26002025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
26012025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silos/{silo}/policy versions = all
26022025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/silos/{silo}/policy versions = all
26032025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/silos/{silo}/quotas versions = all
26042025-04-30T06:46:05.833ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
26052025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/system/timeseries/query versions = all
26062025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/timeseries/schemas versions = all
26072025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/update/repository versions = all
26082025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/update/repository/{system_version} versions = all
26092025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/update/target-release versions = all
26102025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/system/update/target-release versions = all
26112025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/users versions = all
26122025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/users/{user_id} versions = all
26132025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/users-builtin versions = all
26142025-04-30T06:46:05.834ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/users-builtin/{user} versions = all
26152025-04-30T06:46:05.835ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/utilization/silos versions = all
26162025-04-30T06:46:05.835ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/system/utilization/silos/{silo} versions = all
26172025-04-30T06:46:05.835ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/timeseries/query versions = all
26182025-04-30T06:46:05.835ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/users versions = all
26192025-04-30T06:46:05.835ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/utilization versions = all
26202025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-firewall-rules versions = all
26212025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/vpc-firewall-rules versions = all
26222025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-router-routes versions = all
26232025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/vpc-router-routes versions = all
26242025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26252025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-router-routes/{route} versions = all
26262025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26272025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-routers versions = all
26282025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/vpc-routers versions = all
26292025-04-30T06:46:05.836ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/vpc-routers/{router} versions = all
26302025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-routers/{router} versions = all
26312025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/vpc-routers/{router} versions = all
26322025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-subnets versions = all
26332025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/vpc-subnets versions = all
26342025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26352025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26362025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26372025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26382025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpcs versions = all
26392025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/vpcs versions = all
26402025-04-30T06:46:05.837ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/vpcs/{vpc} versions = all
26412025-04-30T06:46:05.838ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/vpcs/{vpc} versions = all
26422025-04-30T06:46:05.838ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/vpcs/{vpc} versions = all
26432025-04-30T06:46:05.838ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/webhooks/deliveries versions = all
26442025-04-30T06:46:05.838ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26452025-04-30T06:46:05.838ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/webhooks/event-classes versions = all
26462025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/webhooks/receivers versions = all
26472025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/webhooks/receivers versions = all
26482025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26492025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26502025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26512025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26522025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26532025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26542025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = GET path = /v1/webhooks/secrets versions = all
26552025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = POST path = /v1/webhooks/secrets versions = all
26562025-04-30T06:46:05.839ZDEBGtest_multiple_snapshots (dropshot_external): registered endpoint local_addr = 127.0.0.1:54124 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26572025-04-30T06:46:05.839ZINFOtest_multiple_snapshots (dropshot_external): listening local_addr = 127.0.0.1:54124
26582025-04-30T06:46:05.840ZDEBGtest_multiple_snapshots (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:54124
26592025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = / versions = all
26602025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /access versions = all
26612025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /assets/{path} versions = all
26622025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /device/auth versions = all
26632025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /device/confirm versions = all
26642025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /device/success versions = all
26652025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /device/token versions = all
26662025-04-30T06:46:05.900ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /device/verify versions = all
26672025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/probes versions = all
26682025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /experimental/v1/probes versions = all
26692025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26702025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/probes/{probe} versions = all
26712025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/system/support-bundles versions = all
26722025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /experimental/v1/system/support-bundles versions = all
26732025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26742025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26752025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26762025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26772025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26782025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26792025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26802025-04-30T06:46:05.901ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /images versions = all
26812025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /login versions = all
26822025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /login/{silo_name}/local versions = all
26832025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26842025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26852025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26862025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /lookup/{path} versions = all
26872025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /projects/{path} versions = all
26882025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /projects-new versions = all
26892025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /settings/{path} versions = all
26902025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /system/{path} versions = all
26912025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /utilization versions = all
26922025-04-30T06:46:05.902ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/affinity-groups versions = all
26932025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/affinity-groups versions = all
26942025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26952025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26962025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26972025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26982025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26992025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27002025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27012025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/anti-affinity-groups versions = all
27022025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/anti-affinity-groups versions = all
27032025-04-30T06:46:05.903ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27042025-04-30T06:46:05.904ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27052025-04-30T06:46:05.904ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27062025-04-30T06:46:05.905ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
27072025-04-30T06:46:05.905ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27082025-04-30T06:46:05.905ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27092025-04-30T06:46:05.905ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27102025-04-30T06:46:05.905ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/certificates versions = all
27112025-04-30T06:46:05.906ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/certificates versions = all
27122025-04-30T06:46:05.906ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/certificates/{certificate} versions = all
27132025-04-30T06:46:05.906ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/certificates/{certificate} versions = all
27142025-04-30T06:46:05.906ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/disks versions = all
27152025-04-30T06:46:05.906ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/disks versions = all
27162025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/disks/{disk} versions = all
27172025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/disks/{disk} versions = all
27182025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/disks/{disk}/bulk-write versions = all
27192025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27202025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27212025-04-30T06:46:05.907ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/disks/{disk}/finalize versions = all
27222025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27232025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/floating-ips versions = all
27242025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/floating-ips versions = all
27252025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27262025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27272025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27282025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27292025-04-30T06:46:05.908ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27302025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/groups versions = all
27312025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/groups/{group_id} versions = all
27322025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/images versions = all
27332025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/images versions = all
27342025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/images/{image} versions = all
27352025-04-30T06:46:05.909ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/images/{image} versions = all
27362025-04-30T06:46:05.910ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/images/{image}/demote versions = all
27372025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/images/{image}/promote versions = all
27382025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances versions = all
27392025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances versions = all
27402025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/instances/{instance} versions = all
27412025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance} versions = all
27422025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/instances/{instance} versions = all
27432025-04-30T06:46:05.911ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27442025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27452025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/disks versions = all
27462025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27472025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27482025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/external-ips versions = all
27492025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27502025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27512025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/reboot versions = all
27522025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/serial-console versions = all
27532025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27542025-04-30T06:46:05.912ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27552025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/start versions = all
27562025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/instances/{instance}/stop versions = all
27572025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27582025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27592025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27602025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/internet-gateway-ip-pools versions = all
27612025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/internet-gateway-ip-pools versions = all
27622025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27632025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/internet-gateways versions = all
27642025-04-30T06:46:05.913ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/internet-gateways versions = all
27652025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27662025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/internet-gateways/{gateway} versions = all
27672025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/ip-pools versions = all
27682025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/ip-pools/{pool} versions = all
27692025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/login/{silo_name}/local versions = all
27702025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/logout versions = all
27712025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/me versions = all
27722025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/me/groups versions = all
27732025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/me/ssh-keys versions = all
27742025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/me/ssh-keys versions = all
27752025-04-30T06:46:05.914ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27762025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27772025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/metrics/{metric_name} versions = all
27782025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/network-interfaces versions = all
27792025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/network-interfaces versions = all
27802025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27812025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/network-interfaces/{interface} versions = all
27822025-04-30T06:46:05.915ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/network-interfaces/{interface} versions = all
27832025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/ping versions = all
27842025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/policy versions = all
27852025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/policy versions = all
27862025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/projects versions = all
27872025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/projects versions = all
27882025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/projects/{project} versions = all
27892025-04-30T06:46:05.916ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/projects/{project} versions = all
27902025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/projects/{project} versions = all
27912025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/projects/{project}/policy versions = all
27922025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/projects/{project}/policy versions = all
27932025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/snapshots versions = all
27942025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/snapshots versions = all
27952025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27962025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/snapshots/{snapshot} versions = all
27972025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/disks versions = all
27982025-04-30T06:46:05.917ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27992025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
28002025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/racks versions = all
28012025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
28022025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/sleds versions = all
28032025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/hardware/sleds versions = all
28042025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
28052025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
28062025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
28072025-04-30T06:46:05.918ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
28082025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
28092025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/switch-port versions = all
28102025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28112025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28122025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
28132025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
28142025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
28152025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/switches versions = all
28162025-04-30T06:46:05.919ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
28172025-04-30T06:46:05.920ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/identity-providers versions = all
28182025-04-30T06:46:05.920ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/identity-providers/local/users versions = all
28192025-04-30T06:46:05.920ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28202025-04-30T06:46:05.920ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28212025-04-30T06:46:05.920ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/identity-providers/saml versions = all
28222025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28232025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools versions = all
28242025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools versions = all
28252025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28262025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools/{pool} versions = all
28272025-04-30T06:46:05.921ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28282025-04-30T06:46:05.922ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28292025-04-30T06:46:05.922ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28302025-04-30T06:46:05.922ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28312025-04-30T06:46:05.922ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28322025-04-30T06:46:05.923ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28332025-04-30T06:46:05.923ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28342025-04-30T06:46:05.923ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28352025-04-30T06:46:05.923ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28362025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools-service versions = all
28372025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28382025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28392025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28402025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/metrics/{metric_name} versions = all
28412025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/address-lot versions = all
28422025-04-30T06:46:05.924ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/address-lot versions = all
28432025-04-30T06:46:05.925ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28442025-04-30T06:46:05.925ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28452025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/allow-list versions = all
28462025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/networking/allow-list versions = all
28472025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/bfd-disable versions = all
28482025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/bfd-enable versions = all
28492025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bfd-status versions = all
28502025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/networking/bgp versions = all
28512025-04-30T06:46:05.926ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp versions = all
28522025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/bgp versions = all
28532025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28542025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28552025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28562025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28572025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-exported versions = all
28582025-04-30T06:46:05.927ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-message-history versions = all
28592025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28602025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/bgp-status versions = all
28612025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/loopback-address versions = all
28622025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/loopback-address versions = all
28632025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28642025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28652025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/switch-port-settings versions = all
28662025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/networking/switch-port-settings versions = all
28672025-04-30T06:46:05.928ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28682025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/policy versions = all
28692025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/policy versions = all
28702025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/roles versions = all
28712025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/roles/{role_name} versions = all
28722025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silo-quotas versions = all
28732025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silos versions = all
28742025-04-30T06:46:05.929ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/silos versions = all
28752025-04-30T06:46:05.930ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/system/silos/{silo} versions = all
28762025-04-30T06:46:05.930ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silos/{silo} versions = all
28772025-04-30T06:46:05.930ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28782025-04-30T06:46:05.930ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silos/{silo}/policy versions = all
28792025-04-30T06:46:05.930ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28802025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28812025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28822025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/system/timeseries/query versions = all
28832025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/timeseries/schemas versions = all
28842025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/update/repository versions = all
28852025-04-30T06:46:05.931ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/update/repository/{system_version} versions = all
28862025-04-30T06:46:05.932ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/update/target-release versions = all
28872025-04-30T06:46:05.932ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/system/update/target-release versions = all
28882025-04-30T06:46:05.932ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/users versions = all
28892025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/users/{user_id} versions = all
28902025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/users-builtin versions = all
28912025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/users-builtin/{user} versions = all
28922025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/utilization/silos versions = all
28932025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28942025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/timeseries/query versions = all
28952025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/users versions = all
28962025-04-30T06:46:05.933ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/utilization versions = all
28972025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-firewall-rules versions = all
28982025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/vpc-firewall-rules versions = all
28992025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-router-routes versions = all
29002025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/vpc-router-routes versions = all
29012025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
29022025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-router-routes/{route} versions = all
29032025-04-30T06:46:05.934ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/vpc-router-routes/{route} versions = all
29042025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-routers versions = all
29052025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/vpc-routers versions = all
29062025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/vpc-routers/{router} versions = all
29072025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-routers/{router} versions = all
29082025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/vpc-routers/{router} versions = all
29092025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-subnets versions = all
29102025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/vpc-subnets versions = all
29112025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
29122025-04-30T06:46:05.935ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-subnets/{subnet} versions = all
29132025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
29142025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
29152025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpcs versions = all
29162025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/vpcs versions = all
29172025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/vpcs/{vpc} versions = all
29182025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/vpcs/{vpc} versions = all
29192025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/vpcs/{vpc} versions = all
29202025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/webhooks/deliveries versions = all
29212025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29222025-04-30T06:46:05.936ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/webhooks/event-classes versions = all
29232025-04-30T06:46:05.937ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/webhooks/receivers versions = all
29242025-04-30T06:46:05.937ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/webhooks/receivers versions = all
29252025-04-30T06:46:05.937ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29262025-04-30T06:46:05.937ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29272025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29282025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29292025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29302025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29312025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = GET path = /v1/webhooks/secrets versions = all
29322025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = POST path = /v1/webhooks/secrets versions = all
29332025-04-30T06:46:05.938ZDEBGtest_multiple_snapshots (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:62536 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29342025-04-30T06:46:05.939ZINFOtest_multiple_snapshots (dropshot_external_techport): listening local_addr = 127.0.0.1:62536
29352025-04-30T06:46:05.940ZDEBGtest_multiple_snapshots (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:62536
29362025-04-30T06:46:05.941ZDEBGtest_multiple_snapshots (nexus-producer-server): registered DTrace probes
29372025-04-30T06:46:05.942ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = 127.0.0.1:51104 method = GET path = /{producer_id} versions = all
29382025-04-30T06:46:05.942ZINFOtest_multiple_snapshots (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:51104
29392025-04-30T06:46:05.958ZDEBGtest_multiple_snapshots (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:51104
29402025-04-30T06:46:05.958ZDEBGtest_multiple_snapshots (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:51104
29412025-04-30T06:46:05.958ZDEBGtest_multiple_snapshots (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:55015
29422025-04-30T06:46:05.958ZDEBGtest_multiple_snapshots (nexus-producer-server): starting producer registration task
29432025-04-30T06:46:05.959ZINFOtest_multiple_snapshots (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:51104 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
29442025-04-30T06:46:05.959ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_oximeter
29452025-04-30T06:46:05.959ZDEBGtest_multiple_snapshots: Starting Oximeter
29462025-04-30T06:46:05.960ZDEBGtest_multiple_snapshots (oximeter): registered DTrace probes
29472025-04-30T06:46:05.960ZINFOtest_multiple_snapshots (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29482025-04-30T06:46:05.960ZDEBGtest_multiple_snapshots (oximeter): creating ClickHouse client
29492025-04-30T06:46:05.962ZDEBGtest_multiple_snapshots (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29502025-04-30T06:46:05.962ZDEBGtest_multiple_snapshots (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29512025-04-30T06:46:05.962ZDEBGtest_multiple_snapshots (producer-registration-task): will register with Nexus at 127.0.0.1:55015
29522025-04-30T06:46:06.026ZDEBGtest_multiple_snapshots (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:55015/metrics/producers
29532025-04-30T06:46:06.028ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:35246
29542025-04-30T06:46:06.032ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35246 req_id = 1378ed34-4c0d-4b4f-825f-6590e4594433 uri = /metrics/producers
29552025-04-30T06:46:06.033ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35246 req_id = 1378ed34-4c0d-4b4f-825f-6590e4594433 resource = Database uri = /metrics/producers
29562025-04-30T06:46:06.033ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35246 req_id = 1378ed34-4c0d-4b4f-825f-6590e4594433 roles = RoleSet { roles: {} } uri = /metrics/producers
29572025-04-30T06:46:06.035ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35246 req_id = 1378ed34-4c0d-4b4f-825f-6590e4594433 resource = Database result = Ok(()) uri = /metrics/producers
29582025-04-30T06:46:06.109ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
29592025-04-30T06:46:06.109ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
29602025-04-30T06:46:06.110ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29612025-04-30T06:46:06.113ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:55232 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29622025-04-30T06:46:06.118ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:55232 req_id = 04c3b6b7-aa7f-4e4e-90e2-4e8a38fa3717 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29632025-04-30T06:46:06.120ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2019 local_addr = [::1]:54590 method = GET remote_addr = [::1]:55232 req_id = 04c3b6b7-aa7f-4e4e-90e2-4e8a38fa3717 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29642025-04-30T06:46:06.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "04c3b6b7-aa7f-4e4e-90e2-4e8a38fa3717", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:46:06 GMT"} })
29652025-04-30T06:46:06.122ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29662025-04-30T06:46:06.122ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29672025-04-30T06:46:06.122ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
29682025-04-30T06:46:06.125ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
29692025-04-30T06:46:06.125ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29702025-04-30T06:46:06.125ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29712025-04-30T06:46:06.125ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29722025-04-30T06:46:06.126ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29732025-04-30T06:46:06.126ZWARNtest_multiple_snapshots (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29742025-04-30T06:46:06.126ZDEBGtest_multiple_snapshots (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29752025-04-30T06:46:06.126ZINFOtest_multiple_snapshots (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29762025-04-30T06:46:06.127ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29772025-04-30T06:46:06.129ZWARNtest_multiple_snapshots (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29782025-04-30T06:46:06.129ZINFOtest_multiple_snapshots (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = df13fd92-fb86-423f-8ce8-7026f1aca352 version = 0
29792025-04-30T06:46:06.129ZINFOtest_multiple_snapshots (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29802025-04-30T06:46:06.130ZDEBGtest_multiple_snapshots (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
29812025-04-30T06:46:06.131ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29822025-04-30T06:46:06.135ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
29832025-04-30T06:46:06.135ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29842025-04-30T06:46:06.135ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29852025-04-30T06:46:06.136ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29862025-04-30T06:46:06.137ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = CREATE DATABASE IF NOT EXISTS oximeter
29872025-04-30T06:46:06.140ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29882025-04-30T06:46:06.153ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29892025-04-30T06:46:06.154ZINFOtest_multiple_snapshots (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 122207 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35246 req_id = 1378ed34-4c0d-4b4f-825f-6590e4594433 response_code = 503 uri = /metrics/producers
29902025-04-30T06:46:06.155ZDEBGtest_multiple_snapshots (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "1378ed34-4c0d-4b4f-825f-6590e4594433", "content-length": "133", "date": "Wed, 30 Apr 2025 06:46:06 GMT"} })
29912025-04-30T06:46:06.160ZWARNtest_multiple_snapshots (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 197.418504ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"1378ed34-4c0d-4b4f-825f-6590e4594433\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:46:06 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"1378ed34-4c0d-4b4f-825f-6590e4594433\\" }" file = oximeter/producer/src/lib.rs:371
29922025-04-30T06:46:06.162ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29932025-04-30T06:46:06.166ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29942025-04-30T06:46:06.170ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29952025-04-30T06:46:06.180ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29962025-04-30T06:46:06.186ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29972025-04-30T06:46:06.189ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29982025-04-30T06:46:06.191ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29992025-04-30T06:46:06.194ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30002025-04-30T06:46:06.198ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30012025-04-30T06:46:06.201ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30022025-04-30T06:46:06.209ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30032025-04-30T06:46:06.216ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30042025-04-30T06:46:06.229ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30052025-04-30T06:46:06.232ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30062025-04-30T06:46:06.235ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30072025-04-30T06:46:06.243ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30082025-04-30T06:46:06.248ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30092025-04-30T06:46:06.253ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30102025-04-30T06:46:06.260ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30112025-04-30T06:46:06.265ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30122025-04-30T06:46:06.269ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30132025-04-30T06:46:06.270ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30142025-04-30T06:46:06.270ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.521938092s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30152025-04-30T06:46:06.270ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30162025-04-30T06:46:06.274ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30172025-04-30T06:46:06.279ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30182025-04-30T06:46:06.280ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 64 d8 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 d............_cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:50506 req_id = 17aed725-6b29-478c-ab66-61e8942667bc
30192025-04-30T06:46:06.280ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25816,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50506 req_id = 17aed725-6b29-478c-ab66-61e8942667bc
30202025-04-30T06:46:06.280ZDEBGtest_multiple_snapshots (store): query key key = _crucible-pantry._tcp
30212025-04-30T06:46:06.280ZDEBGtest_multiple_snapshots (store): query key key = 6229309a-478e-474f-811a-5c520b1df45a.host
30222025-04-30T06:46:06.280ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50506 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 64080, target: Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal") })) }] req_id = 17aed725-6b29-478c-ab66-61e8942667bc
30232025-04-30T06:46:06.281ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 00 60 01 00 00 01 00 00 00 00 00 01 24 36 32 32 .`..........$622\n0010: 39 33 30 39 61 2d 34 37 38 65 2d 34 37 34 66 2d 9309a-478e-474f-\n0020: 38 31 31 61 2d 35 63 35 32 30 62 31 64 66 34 35 811a-5c520b1df45\n0030: 61 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 a.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:64500 req_id = 7f632b22-3c82-4f54-aa09-39d127314ad1
30242025-04-30T06:46:06.281ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 96,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 54,\n 50,\n 50,\n 57,\n 51,\n 48,\n 57,\n 97,\n 45,\n 52,\n 55,\n 56,\n 101,\n 45,\n 52,\n 55,\n 52,\n 102,\n 45,\n 56,\n 49,\n 49,\n 97,\n 45,\n 53,\n 99,\n 53,\n 50,\n 48,\n 98,\n 49,\n 100,\n 102,\n 52,\n 53,\n 97,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64500 req_id = 7f632b22-3c82-4f54-aa09-39d127314ad1
30252025-04-30T06:46:06.281ZDEBGtest_multiple_snapshots (store): query key key = 6229309a-478e-474f-811a-5c520b1df45a.host
30262025-04-30T06:46:06.281ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64500 query = LowerQuery { name: LowerName(Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal.")), original: Query { name: Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("6229309a-478e-474f-811a-5c520b1df45a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7f632b22-3c82-4f54-aa09-39d127314ad1
30272025-04-30T06:46:06.365ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30282025-04-30T06:46:06.368ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:40622 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30292025-04-30T06:46:06.368ZDEBGtest_multiple_snapshots (producer-registration-task): will register with Nexus at 127.0.0.1:55015
30302025-04-30T06:46:06.452ZDEBGtest_multiple_snapshots (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:55015/metrics/producers
30312025-04-30T06:46:06.453ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30322025-04-30T06:46:06.453ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:35695
30332025-04-30T06:46:06.454ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:40622 req_id = 40ae438c-b914-4551-84ee-f8803115db59 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30342025-04-30T06:46:06.456ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 2348 local_addr = [::1]:64080 method = GET remote_addr = [::1]:40622 req_id = 40ae438c-b914-4551-84ee-f8803115db59 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30352025-04-30T06:46:06.458ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35695 req_id = e989d166-fef9-4e69-a2ef-c13e2b7df4f9 uri = /metrics/producers
30362025-04-30T06:46:06.459ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35695 req_id = e989d166-fef9-4e69-a2ef-c13e2b7df4f9 resource = Database uri = /metrics/producers
30372025-04-30T06:46:06.459ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35695 req_id = e989d166-fef9-4e69-a2ef-c13e2b7df4f9 roles = RoleSet { roles: {} } uri = /metrics/producers
30382025-04-30T06:46:06.462ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35695 req_id = e989d166-fef9-4e69-a2ef-c13e2b7df4f9 resource = Database result = Ok(()) uri = /metrics/producers
30392025-04-30T06:46:06.888ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:55042 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30412025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:43151 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30422025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:45982 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30432025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:34729 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30442025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:64694 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30452025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:36248 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30462025-04-30T06:46:06.894ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:37272 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30472025-04-30T06:46:06.901ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30482025-04-30T06:46:06.902ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:55042 req_id = 68d5ce34-dac7-49a7-9926-da5c63e3acdb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30492025-04-30T06:46:06.902ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:43151 req_id = f9eca98d-c2a1-40a0-99ed-fd0487953b81 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30502025-04-30T06:46:06.902ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:45982 req_id = 80da604c-ced9-40bd-979b-030b77bb142c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30512025-04-30T06:46:06.903ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:34729 req_id = 06e4324e-78a7-42cb-839c-32f389022ae6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30522025-04-30T06:46:06.903ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:64694 req_id = bb16b046-669c-45a4-b3fc-2e982ca6c7bb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30532025-04-30T06:46:06.903ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:36248 req_id = 992c06a9-c895-4a77-a924-7084e6c04194 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30542025-04-30T06:46:06.903ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:37272 req_id = 29350284-696e-4c46-b5ea-307f922ce9fd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30552025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1293 local_addr = [::1]:64080 method = GET remote_addr = [::1]:55042 req_id = 68d5ce34-dac7-49a7-9926-da5c63e3acdb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30562025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1218 local_addr = [::1]:64080 method = GET remote_addr = [::1]:43151 req_id = f9eca98d-c2a1-40a0-99ed-fd0487953b81 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30572025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 898 local_addr = [::1]:64080 method = GET remote_addr = [::1]:45982 req_id = 80da604c-ced9-40bd-979b-030b77bb142c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30582025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 817 local_addr = [::1]:64080 method = GET remote_addr = [::1]:34729 req_id = 06e4324e-78a7-42cb-839c-32f389022ae6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30592025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 751 local_addr = [::1]:64080 method = GET remote_addr = [::1]:64694 req_id = bb16b046-669c-45a4-b3fc-2e982ca6c7bb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30602025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 669 local_addr = [::1]:64080 method = GET remote_addr = [::1]:36248 req_id = 992c06a9-c895-4a77-a924-7084e6c04194 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30612025-04-30T06:46:06.903ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 589 local_addr = [::1]:64080 method = GET remote_addr = [::1]:37272 req_id = 29350284-696e-4c46-b5ea-307f922ce9fd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30622025-04-30T06:46:06.912ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30632025-04-30T06:46:06.918ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30642025-04-30T06:46:06.922ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:46:06.930ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:46:06.934ZINFOtest_multiple_snapshots (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 476841 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:35695 req_id = e989d166-fef9-4e69-a2ef-c13e2b7df4f9 response_code = 503 uri = /metrics/producers
30672025-04-30T06:46:06.935ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:46:06.935ZDEBGtest_multiple_snapshots (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "e989d166-fef9-4e69-a2ef-c13e2b7df4f9", "content-length": "133", "date": "Wed, 30 Apr 2025 06:46:06 GMT"} })
30692025-04-30T06:46:06.937ZWARNtest_multiple_snapshots (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 974.494186ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"e989d166-fef9-4e69-a2ef-c13e2b7df4f9\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:46:06 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"e989d166-fef9-4e69-a2ef-c13e2b7df4f9\\" }" file = oximeter/producer/src/lib.rs:371
30702025-04-30T06:46:06.939ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:46:06.950ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:46:06.955ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:46:06.958ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:46:06.961ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:46:06.964ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:46:06.966ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:46:06.969ZINFOtest_multiple_snapshots (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = df13fd92-fb86-423f-8ce8-7026f1aca352 version = 14
30782025-04-30T06:46:06.969ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:46:06.975ZDEBGtest_multiple_snapshots (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 479c2db1-f17d-4c0e-9dbc-0d353f5f2ce8
30802025-04-30T06:46:06.976ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:56871 method = GET path = /info versions = all
30812025-04-30T06:46:06.976ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:56871 method = GET path = /producers versions = all
30822025-04-30T06:46:06.976ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:56871 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:46:06.976ZDEBGtest_multiple_snapshots (dropshot): registered endpoint local_addr = [::1]:56871 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:46:06.976ZINFOtest_multiple_snapshots (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:56871
30852025-04-30T06:46:06.977ZDEBGtest_multiple_snapshots (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:56871
30862025-04-30T06:46:06.978ZDEBGtest_multiple_snapshots (oximeter): registered USDT probes
30872025-04-30T06:46:06.978ZDEBGtest_multiple_snapshots (oximeter): contacting nexus
30882025-04-30T06:46:07.501ZDEBGtest_multiple_snapshots (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:55015/metrics/collectors
30892025-04-30T06:46:07.509ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:49496
30902025-04-30T06:46:07.511ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:49496 req_id = b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0 uri = /metrics/collectors
30912025-04-30T06:46:07.512ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:49496 req_id = b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0 resource = Database uri = /metrics/collectors
30922025-04-30T06:46:07.512ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:49496 req_id = b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0 roles = RoleSet { roles: {} } uri = /metrics/collectors
30932025-04-30T06:46:07.513ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:49496 req_id = b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0 resource = Database result = Ok(()) uri = /metrics/collectors
30942025-04-30T06:46:07.539ZINFOtest_multiple_snapshots (nexus): registered new oximeter metric collection server address = [::1]:56871 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
30952025-04-30T06:46:07.540ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 28265 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:49496 req_id = b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0 response_code = 204 uri = /metrics/collectors
30962025-04-30T06:46:07.540ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors", status: 204, headers: {"x-request-id": "b3d9f5a3-ea9e-4bca-b0bc-c04cb45b54f0", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
30972025-04-30T06:46:07.540ZINFOtest_multiple_snapshots (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
30982025-04-30T06:46:07.540ZDEBGtest_multiple_snapshots (init_with_steps): Running step start_producer_server
30992025-04-30T06:46:07.541ZDEBGtest_multiple_snapshots: Starting test metric Producer Server
31002025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31012025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31022025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31032025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31042025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31052025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31062025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31072025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31082025-04-30T06:46:07.544ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:55682 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31092025-04-30T06:46:07.544ZINFOtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:55682 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31102025-04-30T06:46:07.545ZDEBGtest_multiple_snapshots (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:55682 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31112025-04-30T06:46:07.545ZINFOtest_multiple_snapshots (SledAgent): Created Simulated Crucible Server address = [::1]:55682 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31122025-04-30T06:46:07.545ZINFOtest_multiple_snapshots (nexus): inserting test physical disk model = test-model physical_disk_id = d414ebdf-bdf6-47bc-b431-d8f6fe761291 serial = totally-unique-serial: d414ebdf-bdf6-47bc-b431-d8f6fe761291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31132025-04-30T06:46:07.546ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31142025-04-30T06:46:07.546ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31152025-04-30T06:46:07.546ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31162025-04-30T06:46:07.547ZINFOtest_multiple_snapshots (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31172025-04-30T06:46:07.615ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:53682
31182025-04-30T06:46:07.616ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31192025-04-30T06:46:07.616ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31202025-04-30T06:46:07.617ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = b1137cab-18b9-4f5b-af6c-09297cc21d2f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31212025-04-30T06:46:07.617ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:53682 req_id = 57de9001-c0d2-4540-a5af-a25bc3f84ea4 uri = /metrics/producers
31222025-04-30T06:46:07.619ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = b1137cab-18b9-4f5b-af6c-09297cc21d2f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31232025-04-30T06:46:07.619ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = b1137cab-18b9-4f5b-af6c-09297cc21d2f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31242025-04-30T06:46:07.620ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = b1137cab-18b9-4f5b-af6c-09297cc21d2f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31252025-04-30T06:46:07.620ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:53682 req_id = 57de9001-c0d2-4540-a5af-a25bc3f84ea4 resource = Database uri = /metrics/producers
31262025-04-30T06:46:07.620ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:53682 req_id = 57de9001-c0d2-4540-a5af-a25bc3f84ea4 roles = RoleSet { roles: {} } uri = /metrics/producers
31272025-04-30T06:46:07.621ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:53682 req_id = 57de9001-c0d2-4540-a5af-a25bc3f84ea4 resource = Database result = Ok(()) uri = /metrics/producers
31282025-04-30T06:46:07.621ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31292025-04-30T06:46:07.621ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31302025-04-30T06:46:07.622ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31312025-04-30T06:46:07.630ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 12768 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = b1137cab-18b9-4f5b-af6c-09297cc21d2f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31322025-04-30T06:46:07.630ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31332025-04-30T06:46:07.630ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31342025-04-30T06:46:07.630ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31352025-04-30T06:46:07.630ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31362025-04-30T06:46:07.631ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31372025-04-30T06:46:07.631ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1137cab-18b9-4f5b-af6c-09297cc21d2f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
31382025-04-30T06:46:07.631ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31392025-04-30T06:46:07.635ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31402025-04-30T06:46:07.640ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31412025-04-30T06:46:07.656ZDEBGtest_multiple_snapshots (producer-registration-task): will register with Nexus at 127.0.0.1:55015
31422025-04-30T06:46:07.781ZDEBGtest_multiple_snapshots (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:55015/metrics/producers
31432025-04-30T06:46:07.781ZINFOtest_multiple_snapshots (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31442025-04-30T06:46:07.782ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 164594 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:53682 req_id = 57de9001-c0d2-4540-a5af-a25bc3f84ea4 response_code = 201 uri = /metrics/producers
31452025-04-30T06:46:07.782ZINFOtest_multiple_snapshots (dropshot_internal): accepted connection local_addr = 127.0.0.1:55015 remote_addr = 127.0.0.1:37773
31462025-04-30T06:46:07.787ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:37773 req_id = 854e40a6-855b-4610-b86e-d4174e36e13a uri = /metrics/producers
31472025-04-30T06:46:07.787ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:37773 req_id = 854e40a6-855b-4610-b86e-d4174e36e13a resource = Database uri = /metrics/producers
31482025-04-30T06:46:07.787ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:37773 req_id = 854e40a6-855b-4610-b86e-d4174e36e13a roles = RoleSet { roles: {} } uri = /metrics/producers
31492025-04-30T06:46:07.788ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:37773 req_id = 854e40a6-855b-4610-b86e-d4174e36e13a resource = Database result = Ok(()) uri = /metrics/producers
31502025-04-30T06:46:07.869ZINFOtest_multiple_snapshots (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
31512025-04-30T06:46:07.869ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 82482 local_addr = 127.0.0.1:55015 method = POST remote_addr = 127.0.0.1:37773 req_id = 854e40a6-855b-4610-b86e-d4174e36e13a response_code = 201 uri = /metrics/producers
31522025-04-30T06:46:07.870ZDEBGtest_multiple_snapshots (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "854e40a6-855b-4610-b86e-d4174e36e13a", "content-length": "41", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
31532025-04-30T06:46:07.871ZDEBGtest_multiple_snapshots (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
31542025-04-30T06:46:07.872ZINFOtest_multiple_snapshots (nexus): upserting zpool physical_disk_id = d414ebdf-bdf6-47bc-b431-d8f6fe761291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 56960c90-a54a-4138-a1e0-18c4bebea18b
31552025-04-30T06:46:07.872ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31562025-04-30T06:46:07.872ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31572025-04-30T06:46:07.873ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31582025-04-30T06:46:07.882ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31592025-04-30T06:46:07.882ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31602025-04-30T06:46:07.882ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31612025-04-30T06:46:07.882ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31622025-04-30T06:46:07.883ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31632025-04-30T06:46:07.885ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31642025-04-30T06:46:07.892ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31652025-04-30T06:46:07.909ZINFOtest_multiple_snapshots (nexus): upserting Crucible dataset dataset_id = 8cccb92e-3532-4b9c-b8e7-7e02d53cab80 zpool_id = 56960c90-a54a-4138-a1e0-18c4bebea18b
31662025-04-30T06:46:07.955ZINFOtest_multiple_snapshots (nexus): inserting test physical disk model = test-model physical_disk_id = d414ebdf-bdf6-47bc-b431-d8f6fe761291 serial = totally-unique-serial: d414ebdf-bdf6-47bc-b431-d8f6fe761291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31672025-04-30T06:46:07.955ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31682025-04-30T06:46:07.955ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31692025-04-30T06:46:07.956ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31702025-04-30T06:46:07.962ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: d414ebdf-bdf6-47bc-b431-d8f6fe761291 (physical_disk), lookup_type: ById(d414ebdf-bdf6-47bc-b431-d8f6fe761291) }
31712025-04-30T06:46:07.962ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31722025-04-30T06:46:07.962ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31732025-04-30T06:46:07.962ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31742025-04-30T06:46:07.963ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31752025-04-30T06:46:07.982ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31762025-04-30T06:46:07.992ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: d414ebdf-bdf6-47bc-b431-d8f6fe761291 (physical_disk), lookup_type: ById(d414ebdf-bdf6-47bc-b431-d8f6fe761291) } result = Ok(())
31772025-04-30T06:46:07.992ZINFOtest_multiple_snapshots (nexus): upserting zpool physical_disk_id = d414ebdf-bdf6-47bc-b431-d8f6fe761291 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 56960c90-a54a-4138-a1e0-18c4bebea18b
31782025-04-30T06:46:07.992ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31792025-04-30T06:46:07.992ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31802025-04-30T06:46:07.992ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31812025-04-30T06:46:07.992ZTRACtest_multiple_snapshots (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31822025-04-30T06:46:07.992ZTRACtest_multiple_snapshots (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31832025-04-30T06:46:07.995ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31842025-04-30T06:46:07.995ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31852025-04-30T06:46:07.995ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31862025-04-30T06:46:07.995ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31872025-04-30T06:46:07.995ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31882025-04-30T06:46:08.005ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31892025-04-30T06:46:08.009ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31902025-04-30T06:46:08.022ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31912025-04-30T06:46:08.023ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31922025-04-30T06:46:08.023ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31932025-04-30T06:46:08.023ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31942025-04-30T06:46:08.024ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31952025-04-30T06:46:08.045ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31962025-04-30T06:46:08.051ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31972025-04-30T06:46:08.057ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31982025-04-30T06:46:08.057ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
31992025-04-30T06:46:08.058ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32002025-04-30T06:46:08.074ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32012025-04-30T06:46:08.074ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32022025-04-30T06:46:08.075ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32032025-04-30T06:46:08.075ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
32042025-04-30T06:46:08.075ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32052025-04-30T06:46:08.076ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32062025-04-30T06:46:08.076ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32072025-04-30T06:46:08.076ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32082025-04-30T06:46:08.076ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32092025-04-30T06:46:08.076ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32102025-04-30T06:46:08.077ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32112025-04-30T06:46:08.081ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32122025-04-30T06:46:08.086ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32132025-04-30T06:46:08.086ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32142025-04-30T06:46:08.086ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
32152025-04-30T06:46:08.087ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32162025-04-30T06:46:08.089ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32172025-04-30T06:46:08.089ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
32182025-04-30T06:46:08.090ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32192025-04-30T06:46:08.090ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32202025-04-30T06:46:08.095ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32212025-04-30T06:46:08.095ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32222025-04-30T06:46:08.095ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32232025-04-30T06:46:08.096ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32242025-04-30T06:46:08.109ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
32252025-04-30T06:46:08.109ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32262025-04-30T06:46:08.111ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d2 e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60389 req_id = 01340406-00a5-4269-8524-8c907677b1c1
32272025-04-30T06:46:08.111ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60389 req_id = 01340406-00a5-4269-8524-8c907677b1c1
32282025-04-30T06:46:08.111ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
32292025-04-30T06:46:08.112ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32302025-04-30T06:46:08.112ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60389 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 01340406-00a5-4269-8524-8c907677b1c1
32312025-04-30T06:46:08.112ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4432, tv_nsec: 911890822 } })
32322025-04-30T06:46:08.113ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
32332025-04-30T06:46:08.113ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 01 f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61275 req_id = ee7d7913-eb53-4cb5-8a32-2ea63c2c9d58
32342025-04-30T06:46:08.113ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 501,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61275 req_id = ee7d7913-eb53-4cb5-8a32-2ea63c2c9d58
32352025-04-30T06:46:08.115ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32362025-04-30T06:46:08.115ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61275 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ee7d7913-eb53-4cb5-8a32-2ea63c2c9d58
32372025-04-30T06:46:08.179ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32382025-04-30T06:46:08.180ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 00 3d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .=..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53209 req_id = f7d38ad1-4958-4d5f-86fc-22c739868e5e
32392025-04-30T06:46:08.180ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53209 req_id = f7d38ad1-4958-4d5f-86fc-22c739868e5e
32402025-04-30T06:46:08.182ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
32412025-04-30T06:46:08.182ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53209 req_id = f7d38ad1-4958-4d5f-86fc-22c739868e5e
32422025-04-30T06:46:08.182ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
32432025-04-30T06:46:08.182ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
32442025-04-30T06:46:08.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
32452025-04-30T06:46:08.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "024e74e9-ea2d-4630-bd12-32d0f2829a4e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32462025-04-30T06:46:08.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
32472025-04-30T06:46:08.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5fad50f-2c9f-4ffa-8bea-e0556d615b41", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32482025-04-30T06:46:08.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
32492025-04-30T06:46:08.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe62af0f-6393-4fb3-b03d-77d080456674", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32502025-04-30T06:46:08.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
32512025-04-30T06:46:08.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "daaf052a-f4ca-4c05-8c22-97686c2ec722", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32522025-04-30T06:46:08.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
32532025-04-30T06:46:08.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38c52c04-0df1-4754-9ea3-d1772ffc816e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32542025-04-30T06:46:08.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
32552025-04-30T06:46:08.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90fdb77-c640-489a-8def-7307017b4fb9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32562025-04-30T06:46:08.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
32572025-04-30T06:46:08.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5a79b48-c2d3-47d1-ad87-30164d2cff7b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32582025-04-30T06:46:08.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
32592025-04-30T06:46:08.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "610eb9f5-534f-4d00-9b7d-7b75c2c2e63b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32602025-04-30T06:46:08.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
32612025-04-30T06:46:08.207ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d59fc3ee-9153-4f9d-9a31-be6f0246d9a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32622025-04-30T06:46:08.207ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
32632025-04-30T06:46:08.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0519879-5329-4779-bb5d-3ed9550fa484", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32642025-04-30T06:46:08.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
32652025-04-30T06:46:08.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87412a0b-e0de-4eac-a997-9f18838ae5b1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32662025-04-30T06:46:08.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
32672025-04-30T06:46:08.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42206f32-69d5-4b8d-b8da-a2edc8febffe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32682025-04-30T06:46:08.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
32692025-04-30T06:46:08.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3067860e-ac46-4b65-9b63-ac337a6ece4d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32702025-04-30T06:46:08.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
32712025-04-30T06:46:08.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85e57681-fb31-4850-a94d-feeec3f03189", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32722025-04-30T06:46:08.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
32732025-04-30T06:46:08.224ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
32742025-04-30T06:46:08.226ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "923367a9-3fff-445c-8bdb-82c09d463d14", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32752025-04-30T06:46:08.226ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
32762025-04-30T06:46:08.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10d4220e-74f5-4216-9bcc-602caa290b03", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32772025-04-30T06:46:08.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
32782025-04-30T06:46:08.233ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5cf598e-4383-4462-8947-c1e8b8cd291f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32792025-04-30T06:46:08.233ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
32802025-04-30T06:46:08.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e26d49f-3c66-4b4a-9b52-11ee079e84aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32812025-04-30T06:46:08.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
32822025-04-30T06:46:08.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c70f88-9601-437e-8aa7-f4be1cb6e933", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32832025-04-30T06:46:08.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
32842025-04-30T06:46:08.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e43fa2-3b2e-416b-89ea-580bc2c4a387", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32852025-04-30T06:46:08.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
32862025-04-30T06:46:08.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccef781c-5b94-4dc4-bc09-fb70b01d9cc2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32872025-04-30T06:46:08.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
32882025-04-30T06:46:08.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71111f7c-e206-482d-9102-04a58837707e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32892025-04-30T06:46:08.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
32902025-04-30T06:46:08.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88c4dfa3-0181-405b-940b-7a2ad6319625", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32912025-04-30T06:46:08.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
32922025-04-30T06:46:08.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35d8af12-5061-417c-8adc-b103314dcaf2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32932025-04-30T06:46:08.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
32942025-04-30T06:46:08.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0a425d-7188-45b3-994a-b2ec58da78f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32952025-04-30T06:46:08.262ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
32962025-04-30T06:46:08.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7424ff-0632-498a-a851-0c8b2a1a748d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32972025-04-30T06:46:08.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
32982025-04-30T06:46:08.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "660b342a-691e-4a47-a31c-d4a5e06bb86a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
32992025-04-30T06:46:08.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
33002025-04-30T06:46:08.277ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33012025-04-30T06:46:08.277ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33022025-04-30T06:46:08.277ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33032025-04-30T06:46:08.277ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33042025-04-30T06:46:08.278ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33052025-04-30T06:46:08.278ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97722f4f-33fc-4262-8f0e-878b1aa5ed94", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33062025-04-30T06:46:08.278ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
33072025-04-30T06:46:08.281ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a241d67-d368-47db-bb1a-427af184bc69", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33082025-04-30T06:46:08.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
33092025-04-30T06:46:08.282ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33102025-04-30T06:46:08.290ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33112025-04-30T06:46:08.290ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33122025-04-30T06:46:08.290ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33132025-04-30T06:46:08.291ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33142025-04-30T06:46:08.294ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33152025-04-30T06:46:08.294ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33162025-04-30T06:46:08.295ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33172025-04-30T06:46:08.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2556bc-aaff-4450-a7cb-c8f7b3045060", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33182025-04-30T06:46:08.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
33192025-04-30T06:46:08.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e65c7788-fdcf-4e47-984d-dbc2d8e0acf1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33202025-04-30T06:46:08.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
33212025-04-30T06:46:08.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d55b03-7eeb-46c7-b275-47bc1770da2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33222025-04-30T06:46:08.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
33232025-04-30T06:46:08.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "332eed57-8b85-4a1c-a697-87822a8a036b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33242025-04-30T06:46:08.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
33252025-04-30T06:46:08.315ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55ae6830-0d5f-4f3b-9883-62c0f69a7af6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33262025-04-30T06:46:08.315ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
33272025-04-30T06:46:08.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "648f478c-a53c-4ff5-b5c7-b3044ef63174", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33282025-04-30T06:46:08.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
33292025-04-30T06:46:08.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00c72f78-ec8e-4250-9f7b-970f4f0b564c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33302025-04-30T06:46:08.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
33312025-04-30T06:46:08.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1b80f23-121f-4db1-a74e-eacdda7ad094", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33322025-04-30T06:46:08.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
33332025-04-30T06:46:08.337ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "402a7f0f-bab6-4f7c-8df0-addf44645e5a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33342025-04-30T06:46:08.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
33352025-04-30T06:46:08.340ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "381c78d2-bb5d-46b3-829c-8b9ae3c9a0ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33362025-04-30T06:46:08.343ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
33372025-04-30T06:46:08.346ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4590fe2b-44ae-4acd-9699-4d83e32b19d1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33382025-04-30T06:46:08.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
33392025-04-30T06:46:08.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33cb5aee-5767-47cf-bf0a-c6d07c969f6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33402025-04-30T06:46:08.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
33412025-04-30T06:46:08.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b4cdeb-cc7d-45c7-8e31-8dfdde4ffbb1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33422025-04-30T06:46:08.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
33432025-04-30T06:46:08.358ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
33442025-04-30T06:46:08.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58839414-d137-4b63-997f-5e34f0e82cea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33452025-04-30T06:46:08.362ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
33462025-04-30T06:46:08.365ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a450128-d771-45fd-b47a-ab3a90b4e10e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33472025-04-30T06:46:08.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
33482025-04-30T06:46:08.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3ea0997-22c7-4347-9dff-2cf32764d050", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:07 GMT"} })
33492025-04-30T06:46:08.370ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33502025-04-30T06:46:08.370ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33512025-04-30T06:46:08.370ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33522025-04-30T06:46:08.370ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33532025-04-30T06:46:08.371ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33542025-04-30T06:46:08.377ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33552025-04-30T06:46:08.383ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33562025-04-30T06:46:08.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33572025-04-30T06:46:08.383ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33582025-04-30T06:46:08.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33592025-04-30T06:46:08.383ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33602025-04-30T06:46:08.383ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33612025-04-30T06:46:08.398ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33622025-04-30T06:46:08.409ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33632025-04-30T06:46:08.410ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33642025-04-30T06:46:08.410ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33652025-04-30T06:46:08.410ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33662025-04-30T06:46:08.410ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33672025-04-30T06:46:08.411ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33682025-04-30T06:46:08.411ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33692025-04-30T06:46:08.411ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33702025-04-30T06:46:08.412ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33712025-04-30T06:46:08.416ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33722025-04-30T06:46:08.429ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33732025-04-30T06:46:08.429ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33742025-04-30T06:46:08.429ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33752025-04-30T06:46:08.430ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33762025-04-30T06:46:08.513ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
33772025-04-30T06:46:08.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
33782025-04-30T06:46:08.521ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:60113 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33792025-04-30T06:46:08.522ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:60113 req_id = 091b9c4f-8c66-4ccc-81dd-cb21a5fb263c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33802025-04-30T06:46:08.531ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 9266 local_addr = [::1]:54590 method = GET remote_addr = [::1]:60113 req_id = 091b9c4f-8c66-4ccc-81dd-cb21a5fb263c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33812025-04-30T06:46:08.531ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33822025-04-30T06:46:08.531ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
33832025-04-30T06:46:08.532ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33842025-04-30T06:46:08.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "091b9c4f-8c66-4ccc-81dd-cb21a5fb263c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
33852025-04-30T06:46:08.536ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33862025-04-30T06:46:08.537ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33872025-04-30T06:46:08.537ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
33882025-04-30T06:46:08.541ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33892025-04-30T06:46:08.541ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33902025-04-30T06:46:08.541ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33912025-04-30T06:46:08.541ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33922025-04-30T06:46:08.543ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33932025-04-30T06:46:08.557ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33942025-04-30T06:46:08.562ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33952025-04-30T06:46:08.562ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33962025-04-30T06:46:08.562ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33972025-04-30T06:46:08.563ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33982025-04-30T06:46:08.630ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
33992025-04-30T06:46:08.681ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34002025-04-30T06:46:08.681ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34012025-04-30T06:46:08.681ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34022025-04-30T06:46:08.681ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
34032025-04-30T06:46:08.682ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34042025-04-30T06:46:08.685ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34052025-04-30T06:46:08.691ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34062025-04-30T06:46:08.691ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34072025-04-30T06:46:08.691ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
34082025-04-30T06:46:08.691ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34092025-04-30T06:46:08.708ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34102025-04-30T06:46:08.708ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34112025-04-30T06:46:08.709ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 633.006815ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34122025-04-30T06:46:08.709ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34132025-04-30T06:46:08.709ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34142025-04-30T06:46:08.709ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34152025-04-30T06:46:08.709ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34162025-04-30T06:46:08.709ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34172025-04-30T06:46:08.711ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34182025-04-30T06:46:08.711ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34192025-04-30T06:46:08.711ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
34202025-04-30T06:46:08.711ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34212025-04-30T06:46:08.715ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34222025-04-30T06:46:08.718ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34232025-04-30T06:46:08.718ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34242025-04-30T06:46:08.718ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34252025-04-30T06:46:08.719ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34262025-04-30T06:46:08.722ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
34272025-04-30T06:46:08.722ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34282025-04-30T06:46:08.723ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3e c6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 >............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63819 req_id = 852ee31d-c916-481c-9fd8-3ed754495b23
34292025-04-30T06:46:08.723ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16070,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63819 req_id = 852ee31d-c916-481c-9fd8-3ed754495b23
34302025-04-30T06:46:08.723ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
34312025-04-30T06:46:08.723ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34322025-04-30T06:46:08.723ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63819 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 852ee31d-c916-481c-9fd8-3ed754495b23
34332025-04-30T06:46:08.724ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4433, tv_nsec: 523130407 } })
34342025-04-30T06:46:08.724ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
34352025-04-30T06:46:08.724ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 32 a0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 2...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65511 req_id = d47460fe-34a4-4bdc-9cb3-25a006f12ca1
34362025-04-30T06:46:08.724ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12960,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65511 req_id = d47460fe-34a4-4bdc-9cb3-25a006f12ca1
34372025-04-30T06:46:08.724ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34382025-04-30T06:46:08.724ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65511 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d47460fe-34a4-4bdc-9cb3-25a006f12ca1
34392025-04-30T06:46:08.768ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34402025-04-30T06:46:08.769ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3d 61 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c =a..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53530 req_id = 56bfbf6e-f882-44c3-a8c6-5b644f77199b
34412025-04-30T06:46:08.769ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15713,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53530 req_id = 56bfbf6e-f882-44c3-a8c6-5b644f77199b
34422025-04-30T06:46:08.769ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
34432025-04-30T06:46:08.769ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15713, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53530 req_id = 56bfbf6e-f882-44c3-a8c6-5b644f77199b
34442025-04-30T06:46:08.770ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
34452025-04-30T06:46:08.770ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
34462025-04-30T06:46:08.770ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
34472025-04-30T06:46:08.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c3647c1-3257-481a-ae95-a91002d50222", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34482025-04-30T06:46:08.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
34492025-04-30T06:46:08.778ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "284cb4d8-7f01-43b5-bd49-21ef89e245f2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34502025-04-30T06:46:08.778ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
34512025-04-30T06:46:08.781ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2deada8-feb9-4882-8ba1-70eaee09e58b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34522025-04-30T06:46:08.781ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
34532025-04-30T06:46:08.785ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f98266ed-a0bd-44e9-8bdf-bf15c3b6387b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34542025-04-30T06:46:08.785ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
34552025-04-30T06:46:08.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81ced29b-a8e1-464d-8bb0-22d4ef0f48c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34562025-04-30T06:46:08.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
34572025-04-30T06:46:08.819ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
34582025-04-30T06:46:08.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c38a7075-5975-4b41-aeee-c4133ff9d88e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34592025-04-30T06:46:08.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
34602025-04-30T06:46:08.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cbc82ea-660d-43eb-9656-9dbad8d874e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34612025-04-30T06:46:08.826ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
34622025-04-30T06:46:08.828ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccd635eb-0111-41b3-9031-e1e9779371c8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34632025-04-30T06:46:08.828ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
34642025-04-30T06:46:08.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e275b6f-a43f-4f6b-b822-1f92c0fd9c26", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34652025-04-30T06:46:08.830ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
34662025-04-30T06:46:08.833ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "544f7adb-7239-4cd7-b0dd-aaa1604a90bb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34672025-04-30T06:46:08.833ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
34682025-04-30T06:46:08.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01fb5550-ebca-4151-b17e-69f35bb32ae1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34692025-04-30T06:46:08.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
34702025-04-30T06:46:08.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b690d5b4-66e5-4db5-95a4-70b3ca83e1f1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34712025-04-30T06:46:08.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
34722025-04-30T06:46:08.838ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb6a51f-78f1-46c8-bf81-f6479de610c4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34732025-04-30T06:46:08.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
34742025-04-30T06:46:08.843ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3316942d-e223-4229-b301-f21edd554bc0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34752025-04-30T06:46:08.843ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
34762025-04-30T06:46:08.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "624df02e-b31f-48a2-b65d-4dd1c9e8611b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34772025-04-30T06:46:08.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
34782025-04-30T06:46:08.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ada3528-2566-460c-958d-f0384e3a34b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34792025-04-30T06:46:08.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
34802025-04-30T06:46:08.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3922530a-acc4-40a0-b94b-c3faa82e3af4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34812025-04-30T06:46:08.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
34822025-04-30T06:46:08.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea85c485-5f2e-42d6-954d-9539beb7a738", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34832025-04-30T06:46:08.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
34842025-04-30T06:46:08.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea39889-ef0b-4078-ad62-aed6adcc33c7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34852025-04-30T06:46:08.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
34862025-04-30T06:46:08.856ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94197673-5b28-4cfc-8a59-7c14f25e88ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34872025-04-30T06:46:08.856ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
34882025-04-30T06:46:08.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb605e4-d5e6-4669-a185-26b67d16abb3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34892025-04-30T06:46:08.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
34902025-04-30T06:46:08.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0efd79ac-29d0-4331-9f39-e0ca7555f6b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34912025-04-30T06:46:08.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
34922025-04-30T06:46:08.859ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2975d5c7-122a-4ed3-8457-04ce0a980a04", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34932025-04-30T06:46:08.859ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
34942025-04-30T06:46:08.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae9f235-8f82-49bc-8018-8759d3c19dd4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34952025-04-30T06:46:08.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
34962025-04-30T06:46:08.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7825e3f6-3da9-4ce2-a5ea-355041093471", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34972025-04-30T06:46:08.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
34982025-04-30T06:46:08.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dedc28ee-ada6-439d-9b7c-ed46b3804883", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
34992025-04-30T06:46:08.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
35002025-04-30T06:46:08.869ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35012025-04-30T06:46:08.870ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35022025-04-30T06:46:08.870ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35032025-04-30T06:46:08.870ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35042025-04-30T06:46:08.871ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35052025-04-30T06:46:08.876ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35062025-04-30T06:46:08.882ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35072025-04-30T06:46:08.883ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35082025-04-30T06:46:08.883ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35092025-04-30T06:46:08.884ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35102025-04-30T06:46:08.886ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d721dff-2fc5-4852-8ea2-b7e1428cd06a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35112025-04-30T06:46:08.886ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
35122025-04-30T06:46:08.888ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35132025-04-30T06:46:08.888ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35142025-04-30T06:46:08.889ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35152025-04-30T06:46:08.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99dd730-03e3-45d7-bfc9-d4c1e52133c3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35162025-04-30T06:46:08.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
35172025-04-30T06:46:08.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "304dfad2-e712-4575-8e10-33dc9b0f8006", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35182025-04-30T06:46:08.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
35192025-04-30T06:46:08.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5fee799-2fe1-4789-8f0c-03284f73e618", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35202025-04-30T06:46:08.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
35212025-04-30T06:46:08.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "203c6c7b-55d5-4157-a62c-304d4105d9cd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35222025-04-30T06:46:08.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
35232025-04-30T06:46:08.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb202294-0f19-4406-ac63-5abfb780db50", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35242025-04-30T06:46:08.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
35252025-04-30T06:46:08.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bd0a154-20bb-4d7c-87aa-ae14833f588e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35262025-04-30T06:46:08.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
35272025-04-30T06:46:08.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d93fbf-2e0d-414c-b470-adcd4f7e5e53", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35282025-04-30T06:46:08.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
35292025-04-30T06:46:08.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f5a11e1-1e27-4a4f-8e63-c4f4d8e55a48", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35302025-04-30T06:46:08.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
35312025-04-30T06:46:08.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "530edb57-3cc4-4677-8ac8-875f9c9064ca", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35322025-04-30T06:46:08.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
35332025-04-30T06:46:08.927ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
35342025-04-30T06:46:08.947ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efe7ff28-96e0-4fe8-a78c-661fa7815cd9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35352025-04-30T06:46:08.947ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
35362025-04-30T06:46:08.952ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f5d2cef-9e85-4416-8c6a-82b2f081ad1b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35372025-04-30T06:46:08.953ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
35382025-04-30T06:46:08.954ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f14dea-fadb-4c4d-8d19-fe2af970ce45", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35392025-04-30T06:46:08.954ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
35402025-04-30T06:46:08.956ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6bdf322-b307-4f99-b87e-a914cedfd9cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35412025-04-30T06:46:08.956ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
35422025-04-30T06:46:08.958ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "075e6298-1eca-45ba-8653-dc3c84bf9121", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35432025-04-30T06:46:08.958ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
35442025-04-30T06:46:08.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5dd6ae-00d3-4dc6-b9dc-f65db0799b73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35452025-04-30T06:46:08.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
35462025-04-30T06:46:08.960ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c09cd9-a75e-455e-9640-8aaba5cab59d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35472025-04-30T06:46:08.961ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
35482025-04-30T06:46:08.963ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60fac058-b24c-4f43-b4d1-abc1c71274c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35492025-04-30T06:46:08.963ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
35502025-04-30T06:46:08.964ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99dced1a-765c-42b1-a9ad-977e7fae4bb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35512025-04-30T06:46:08.965ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35522025-04-30T06:46:08.965ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35532025-04-30T06:46:08.965ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35542025-04-30T06:46:08.965ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35552025-04-30T06:46:08.966ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35562025-04-30T06:46:08.969ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35572025-04-30T06:46:08.982ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35582025-04-30T06:46:08.982ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35592025-04-30T06:46:08.982ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35602025-04-30T06:46:08.982ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35612025-04-30T06:46:08.982ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35622025-04-30T06:46:08.983ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35632025-04-30T06:46:08.983ZTRACtest_multiple_snapshots (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35642025-04-30T06:46:08.983ZTRACtest_multiple_snapshots (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35652025-04-30T06:46:09.019ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35662025-04-30T06:46:09.019ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35672025-04-30T06:46:09.020ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35682025-04-30T06:46:09.020ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35692025-04-30T06:46:09.020ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35702025-04-30T06:46:09.022ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35712025-04-30T06:46:09.026ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35722025-04-30T06:46:09.026ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35732025-04-30T06:46:09.026ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35742025-04-30T06:46:09.027ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35752025-04-30T06:46:09.035ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35762025-04-30T06:46:09.043ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35772025-04-30T06:46:09.044ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35782025-04-30T06:46:09.048ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35792025-04-30T06:46:09.048ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35802025-04-30T06:46:09.051ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35812025-04-30T06:46:09.051ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
35822025-04-30T06:46:09.052ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35832025-04-30T06:46:09.188ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
35842025-04-30T06:46:09.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
35852025-04-30T06:46:09.189ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:58757 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35862025-04-30T06:46:09.191ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:58757 req_id = 5b424609-92a9-47c9-a159-203de95c16eb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35872025-04-30T06:46:09.192ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 824 local_addr = [::1]:54590 method = GET remote_addr = [::1]:58757 req_id = 5b424609-92a9-47c9-a159-203de95c16eb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35882025-04-30T06:46:09.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b424609-92a9-47c9-a159-203de95c16eb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
35892025-04-30T06:46:09.197ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35902025-04-30T06:46:09.197ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35912025-04-30T06:46:09.197ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
35922025-04-30T06:46:09.199ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
35932025-04-30T06:46:09.199ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35942025-04-30T06:46:09.199ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35952025-04-30T06:46:09.199ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35962025-04-30T06:46:09.200ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35972025-04-30T06:46:09.209ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35982025-04-30T06:46:09.213ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
35992025-04-30T06:46:09.213ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36002025-04-30T06:46:09.213ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36012025-04-30T06:46:09.213ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36022025-04-30T06:46:09.219ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
36032025-04-30T06:46:09.256ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36042025-04-30T06:46:09.257ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36052025-04-30T06:46:09.257ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 548.281394ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36062025-04-30T06:46:09.257ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36072025-04-30T06:46:09.257ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36082025-04-30T06:46:09.257ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36092025-04-30T06:46:09.257ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36102025-04-30T06:46:09.257ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36112025-04-30T06:46:09.258ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36122025-04-30T06:46:09.261ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36132025-04-30T06:46:09.267ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36142025-04-30T06:46:09.267ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36152025-04-30T06:46:09.267ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36162025-04-30T06:46:09.268ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36172025-04-30T06:46:09.272ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36182025-04-30T06:46:09.272ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36192025-04-30T06:46:09.272ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36202025-04-30T06:46:09.272ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36212025-04-30T06:46:09.273ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36222025-04-30T06:46:09.273ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
36232025-04-30T06:46:09.273ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36242025-04-30T06:46:09.277ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 63 e8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 c............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62574 req_id = 9de8fe49-d2ef-4175-a57e-d619a9c24020
36252025-04-30T06:46:09.277ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25576,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62574 req_id = 9de8fe49-d2ef-4175-a57e-d619a9c24020
36262025-04-30T06:46:09.277ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
36272025-04-30T06:46:09.278ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36282025-04-30T06:46:09.278ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62574 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9de8fe49-d2ef-4175-a57e-d619a9c24020
36292025-04-30T06:46:09.279ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36302025-04-30T06:46:09.286ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36312025-04-30T06:46:09.286ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36322025-04-30T06:46:09.286ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36332025-04-30T06:46:09.286ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36342025-04-30T06:46:09.286ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4434, tv_nsec: 85556273 } })
36352025-04-30T06:46:09.286ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
36362025-04-30T06:46:09.287ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f 47 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .G..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56226 req_id = 721ff65f-86e1-432a-82bd-cd2680064194
36372025-04-30T06:46:09.287ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40775,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56226 req_id = 721ff65f-86e1-432a-82bd-cd2680064194
36382025-04-30T06:46:09.287ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36392025-04-30T06:46:09.287ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56226 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 721ff65f-86e1-432a-82bd-cd2680064194
36402025-04-30T06:46:09.352ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36412025-04-30T06:46:09.354ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a8 d2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58178 req_id = 98af3ca5-f109-4d49-9953-ccb63c7f4916
36422025-04-30T06:46:09.354ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43218,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58178 req_id = 98af3ca5-f109-4d49-9953-ccb63c7f4916
36432025-04-30T06:46:09.354ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
36442025-04-30T06:46:09.354ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43218, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58178 req_id = 98af3ca5-f109-4d49-9953-ccb63c7f4916
36452025-04-30T06:46:09.354ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
36462025-04-30T06:46:09.355ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
36472025-04-30T06:46:09.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
36482025-04-30T06:46:09.355ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36492025-04-30T06:46:09.355ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36502025-04-30T06:46:09.355ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36512025-04-30T06:46:09.356ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "da566e70-2102-4ac6-964c-675218ba56af", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36522025-04-30T06:46:09.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
36532025-04-30T06:46:09.358ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56435638-4bae-46b5-af1b-fc2302b89e77", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36542025-04-30T06:46:09.358ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
36552025-04-30T06:46:09.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef4ca23-b358-4e9a-abcf-fd8e9f515577", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36562025-04-30T06:46:09.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
36572025-04-30T06:46:09.363ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f99f5e25-2bd3-40ce-8236-6eaccdac5b66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36582025-04-30T06:46:09.363ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
36592025-04-30T06:46:09.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1691b6-fab7-42c8-a59c-615113787439", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36602025-04-30T06:46:09.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
36612025-04-30T06:46:09.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1e15c00-972a-493b-97f3-921164299ab3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36622025-04-30T06:46:09.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
36632025-04-30T06:46:09.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f461fdc-53cc-4ad5-b271-22eb078a42a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36642025-04-30T06:46:09.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
36652025-04-30T06:46:09.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d9f4748-46c9-4333-b713-071db00bad7d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36662025-04-30T06:46:09.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
36672025-04-30T06:46:09.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8bc0f0d-607f-4cca-afdc-d1ac01a40a81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36682025-04-30T06:46:09.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
36692025-04-30T06:46:09.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f09f3b3-7289-408d-8250-d2c8b269856e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36702025-04-30T06:46:09.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
36712025-04-30T06:46:09.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7b1c07f-bfc0-4f2a-aafd-1fb4c0d955a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36722025-04-30T06:46:09.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
36732025-04-30T06:46:09.398ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c7c223-1e73-4fe6-9eae-32e475f7e6de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36742025-04-30T06:46:09.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
36752025-04-30T06:46:09.402ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
36762025-04-30T06:46:09.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb44d818-1b17-4e87-958d-ae8434cfc5f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36772025-04-30T06:46:09.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
36782025-04-30T06:46:09.407ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ceeab5-7060-4993-b317-58284c202eca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36792025-04-30T06:46:09.407ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
36802025-04-30T06:46:09.461ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36812025-04-30T06:46:09.461ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36822025-04-30T06:46:09.461ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36832025-04-30T06:46:09.462ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36842025-04-30T06:46:09.463ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36852025-04-30T06:46:09.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8681f06b-d532-4fe8-8b97-6f6fb64dc7fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36862025-04-30T06:46:09.467ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
36872025-04-30T06:46:09.486ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36882025-04-30T06:46:09.499ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36892025-04-30T06:46:09.499ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36902025-04-30T06:46:09.499ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36912025-04-30T06:46:09.500ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36922025-04-30T06:46:09.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b584242f-dcbb-44c8-9e5c-2ae88a65b2e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36932025-04-30T06:46:09.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
36942025-04-30T06:46:09.502ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36952025-04-30T06:46:09.503ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
36962025-04-30T06:46:09.503ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36972025-04-30T06:46:09.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50a3bcb7-604c-498e-ac1e-d4783500af6c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
36982025-04-30T06:46:09.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
36992025-04-30T06:46:09.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ce85bb2-d11e-4fa4-8531-c0998e269dc6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37002025-04-30T06:46:09.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
37012025-04-30T06:46:09.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e1aa93-eda5-4f43-90df-929e963df66c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37022025-04-30T06:46:09.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
37032025-04-30T06:46:09.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25bec3fa-12b3-4ac4-90f6-55aed34fa34c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37042025-04-30T06:46:09.511ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
37052025-04-30T06:46:09.511ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a0cd558-76c7-4fdb-bdb7-d4001a5975f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37062025-04-30T06:46:09.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
37072025-04-30T06:46:09.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb6f009-534e-4097-82ae-e04aedb57932", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37082025-04-30T06:46:09.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
37092025-04-30T06:46:09.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c690436-2c35-4288-84ce-d254786ea7f2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37102025-04-30T06:46:09.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
37112025-04-30T06:46:09.515ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc8a5fab-de4d-4c70-aa85-cd3f835c3a70", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37122025-04-30T06:46:09.515ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
37132025-04-30T06:46:09.516ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d799bfa-4ef7-48e3-bcd3-2f8e69254da9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37142025-04-30T06:46:09.517ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
37152025-04-30T06:46:09.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86d48410-e817-44d0-8fd0-cda58540c3a2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:08 GMT"} })
37162025-04-30T06:46:09.520ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
37172025-04-30T06:46:09.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a30a1e9f-8864-478d-9463-133c9d6ab05e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37182025-04-30T06:46:09.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
37192025-04-30T06:46:09.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71b14395-461a-42c8-b115-4c5af684cfdd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37202025-04-30T06:46:09.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
37212025-04-30T06:46:09.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc56d51-00a4-4fee-b9bd-294f5251e04d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37222025-04-30T06:46:09.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
37232025-04-30T06:46:09.534ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
37242025-04-30T06:46:09.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "296783cd-437f-4546-9ccd-dac602db6738", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37252025-04-30T06:46:09.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
37262025-04-30T06:46:09.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f41b0b7d-1743-4f4a-b3fe-e484fd465022", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37272025-04-30T06:46:09.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
37282025-04-30T06:46:09.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9def9932-9db1-40b0-9801-453f71494878", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37292025-04-30T06:46:09.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
37302025-04-30T06:46:09.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa9869a6-6626-4ed2-9ce8-c7576ade2394", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37312025-04-30T06:46:09.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
37322025-04-30T06:46:09.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebfc96cf-da6d-4554-8b29-23b8c4fb58b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37332025-04-30T06:46:09.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
37342025-04-30T06:46:09.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed563a7f-74d0-4ec3-96e3-7100d0eafeab", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37352025-04-30T06:46:09.548ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
37362025-04-30T06:46:09.549ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
37372025-04-30T06:46:09.550ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37382025-04-30T06:46:09.550ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0006331c-28d6-49cf-8d5b-44c1cf6a7add uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37392025-04-30T06:46:09.551ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0006331c-28d6-49cf-8d5b-44c1cf6a7add resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37402025-04-30T06:46:09.551ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0006331c-28d6-49cf-8d5b-44c1cf6a7add roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37412025-04-30T06:46:09.553ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0006331c-28d6-49cf-8d5b-44c1cf6a7add resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37422025-04-30T06:46:09.558ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6994 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0006331c-28d6-49cf-8d5b-44c1cf6a7add response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37432025-04-30T06:46:09.558ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0006331c-28d6-49cf-8d5b-44c1cf6a7add", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37442025-04-30T06:46:09.559ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37452025-04-30T06:46:09.560ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37462025-04-30T06:46:09.561ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37472025-04-30T06:46:09.561ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37482025-04-30T06:46:09.561ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37492025-04-30T06:46:09.564ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6547053-aa31-4bee-b200-bf0e61b5a5cb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37502025-04-30T06:46:09.564ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
37512025-04-30T06:46:09.567ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6884 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37522025-04-30T06:46:09.567ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6c9da54-5f76-4bb9-b46a-e26bf16eb9c7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37532025-04-30T06:46:09.567ZDEBGtest_multiple_snapshots (oximeter-agent): registered new metric producer address = 127.0.0.1:51104 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
37542025-04-30T06:46:09.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55ee4fc9-f941-4f96-ac93-7d28a2df983b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37552025-04-30T06:46:09.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
37562025-04-30T06:46:09.631ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
37572025-04-30T06:46:09.631ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37582025-04-30T06:46:09.631ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37592025-04-30T06:46:09.631ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37602025-04-30T06:46:09.631ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
37612025-04-30T06:46:09.633ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37622025-04-30T06:46:09.634ZDEBGtest_multiple_snapshots (oximeter-agent): registered new metric producer address = [::1]:55661 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37632025-04-30T06:46:09.688ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37642025-04-30T06:46:09.689ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
37652025-04-30T06:46:09.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9426cf79-c672-463f-8c82-830f26fa35ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37662025-04-30T06:46:09.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
37672025-04-30T06:46:09.690ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37682025-04-30T06:46:09.696ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37692025-04-30T06:46:09.696ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37702025-04-30T06:46:09.696ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
37712025-04-30T06:46:09.696ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37722025-04-30T06:46:09.699ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37732025-04-30T06:46:09.699ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
37742025-04-30T06:46:09.699ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37752025-04-30T06:46:09.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fca2629-cd2c-4135-9ae2-6115c303de40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37762025-04-30T06:46:09.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
37772025-04-30T06:46:09.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b64b104d-98a6-4569-a344-189983f10e60", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37782025-04-30T06:46:09.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
37792025-04-30T06:46:09.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4314e244-7200-43fa-9cf4-f285d4840729", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37802025-04-30T06:46:09.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
37812025-04-30T06:46:09.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ff01093-c1db-4ea2-8df7-3af0be8da236", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37822025-04-30T06:46:09.707ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
37832025-04-30T06:46:09.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4df9c3c-1e05-451f-b6af-28175837a217", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37842025-04-30T06:46:09.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
37852025-04-30T06:46:09.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2028edb6-5552-461b-9f91-059868bed297", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37862025-04-30T06:46:09.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
37872025-04-30T06:46:09.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a362745-d687-493f-8c0c-c5d56e84620f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
37882025-04-30T06:46:09.713ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37892025-04-30T06:46:09.713ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37902025-04-30T06:46:09.713ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37912025-04-30T06:46:09.714ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37922025-04-30T06:46:09.715ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37932025-04-30T06:46:09.717ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37942025-04-30T06:46:09.730ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37952025-04-30T06:46:09.730ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37962025-04-30T06:46:09.731ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37972025-04-30T06:46:09.731ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37982025-04-30T06:46:09.731ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37992025-04-30T06:46:09.731ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38002025-04-30T06:46:09.736ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38012025-04-30T06:46:09.747ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38022025-04-30T06:46:09.747ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38032025-04-30T06:46:09.747ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38042025-04-30T06:46:09.749ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38052025-04-30T06:46:09.888ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
38062025-04-30T06:46:09.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
38072025-04-30T06:46:09.889ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:57298 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38082025-04-30T06:46:09.892ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:57298 req_id = 14d04edb-5da9-42f0-b821-39a25b96c740 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38092025-04-30T06:46:09.892ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 589 local_addr = [::1]:54590 method = GET remote_addr = [::1]:57298 req_id = 14d04edb-5da9-42f0-b821-39a25b96c740 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38102025-04-30T06:46:09.893ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d04edb-5da9-42f0-b821-39a25b96c740", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38112025-04-30T06:46:09.894ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38122025-04-30T06:46:09.894ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38132025-04-30T06:46:09.894ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
38142025-04-30T06:46:09.896ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38152025-04-30T06:46:09.897ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38162025-04-30T06:46:09.897ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38172025-04-30T06:46:09.897ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38182025-04-30T06:46:09.897ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38192025-04-30T06:46:09.902ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38202025-04-30T06:46:09.913ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38212025-04-30T06:46:09.913ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38222025-04-30T06:46:09.913ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38232025-04-30T06:46:09.914ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38242025-04-30T06:46:09.915ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
38252025-04-30T06:46:09.966ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38262025-04-30T06:46:09.966ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38272025-04-30T06:46:09.966ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38282025-04-30T06:46:09.966ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
38292025-04-30T06:46:09.967ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38302025-04-30T06:46:09.993ZTRACtest_multiple_snapshots (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38312025-04-30T06:46:09.993ZTRACtest_multiple_snapshots (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38322025-04-30T06:46:09.996ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38332025-04-30T06:46:10.014ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38342025-04-30T06:46:10.014ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38352025-04-30T06:46:10.014ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
38362025-04-30T06:46:10.016ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38372025-04-30T06:46:10.022ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38382025-04-30T06:46:10.025ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38392025-04-30T06:46:10.025ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 766.264472ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38402025-04-30T06:46:10.025ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38412025-04-30T06:46:10.025ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38422025-04-30T06:46:10.025ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38432025-04-30T06:46:10.025ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38442025-04-30T06:46:10.025ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38452025-04-30T06:46:10.025ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38462025-04-30T06:46:10.026ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38472025-04-30T06:46:10.027ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
38482025-04-30T06:46:10.028ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38492025-04-30T06:46:10.029ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38502025-04-30T06:46:10.038ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38512025-04-30T06:46:10.039ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38522025-04-30T06:46:10.039ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38532025-04-30T06:46:10.039ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38542025-04-30T06:46:10.043ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
38552025-04-30T06:46:10.044ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38562025-04-30T06:46:10.045ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9d 89 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50358 req_id = 41691de6-86b4-4d29-9ea7-07d4812f7447
38572025-04-30T06:46:10.045ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40329,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50358 req_id = 41691de6-86b4-4d29-9ea7-07d4812f7447
38582025-04-30T06:46:10.045ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
38592025-04-30T06:46:10.046ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38602025-04-30T06:46:10.046ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50358 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 41691de6-86b4-4d29-9ea7-07d4812f7447
38612025-04-30T06:46:10.046ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4434, tv_nsec: 845361695 } })
38622025-04-30T06:46:10.046ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
38632025-04-30T06:46:10.046ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b8 66 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .f..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56910 req_id = 5214e9b6-d32f-486b-9421-5ccd8f2ad257
38642025-04-30T06:46:10.046ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56910 req_id = 5214e9b6-d32f-486b-9421-5ccd8f2ad257
38652025-04-30T06:46:10.047ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38662025-04-30T06:46:10.047ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56910 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5214e9b6-d32f-486b-9421-5ccd8f2ad257
38672025-04-30T06:46:10.123ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38682025-04-30T06:46:10.124ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3d 46 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c =F..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53202 req_id = 1d7acea3-a739-43a7-a7ea-6f5a4479486a
38692025-04-30T06:46:10.125ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15686,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53202 req_id = 1d7acea3-a739-43a7-a7ea-6f5a4479486a
38702025-04-30T06:46:10.125ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
38712025-04-30T06:46:10.125ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15686, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53202 req_id = 1d7acea3-a739-43a7-a7ea-6f5a4479486a
38722025-04-30T06:46:10.125ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
38732025-04-30T06:46:10.125ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
38742025-04-30T06:46:10.125ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
38752025-04-30T06:46:10.130ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7d61993-9d13-45b5-99e6-f59cf0ae3485", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38762025-04-30T06:46:10.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
38772025-04-30T06:46:10.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3098d44-335b-42a9-b14d-c6bb6c1b646f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38782025-04-30T06:46:10.140ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
38792025-04-30T06:46:10.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a45c0e1-b0d3-4441-9f8a-8cbe56be9990", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38802025-04-30T06:46:10.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
38812025-04-30T06:46:10.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5431b7a-837b-4ce2-86a5-00ee990358ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38822025-04-30T06:46:10.147ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
38832025-04-30T06:46:10.157ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
38842025-04-30T06:46:10.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bcc3fa1-3eae-4fd5-9f94-4a01f858f1cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38852025-04-30T06:46:10.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
38862025-04-30T06:46:10.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abd2163e-acb3-49e7-b32d-c983acf3e3ad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38872025-04-30T06:46:10.161ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
38882025-04-30T06:46:10.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a2655ae-8926-4dca-85f5-caabb3bf3a3b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38892025-04-30T06:46:10.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
38902025-04-30T06:46:10.165ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ffec29d-0d45-48a7-8121-bc6809021a41", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38912025-04-30T06:46:10.165ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
38922025-04-30T06:46:10.166ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0be226bf-1a9d-4a01-8039-b5db63639f18", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38932025-04-30T06:46:10.166ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
38942025-04-30T06:46:10.167ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0abc81cb-7eff-4d35-99c8-8aa95b3c7c99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38952025-04-30T06:46:10.167ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
38962025-04-30T06:46:10.169ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07939867-559f-4329-ac53-619afcad74c9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38972025-04-30T06:46:10.169ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
38982025-04-30T06:46:10.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ec39b9-a276-41f3-9891-2ac5aec640e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
38992025-04-30T06:46:10.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
39002025-04-30T06:46:10.173ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2469393f-fc7f-46fb-947f-5c7137fe65a6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39012025-04-30T06:46:10.173ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
39022025-04-30T06:46:10.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5504d354-3355-40a1-a7f0-de791c16a03a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39032025-04-30T06:46:10.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
39042025-04-30T06:46:10.179ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f71f51-4512-4f32-b5cc-8da8e55b10f0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39052025-04-30T06:46:10.179ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
39062025-04-30T06:46:10.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dc0b408-ef82-4a6a-85f5-bad697a79102", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39072025-04-30T06:46:10.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
39082025-04-30T06:46:10.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec2c996-b4b4-4eee-9839-55994e142132", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39092025-04-30T06:46:10.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
39102025-04-30T06:46:10.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f73f293c-3815-4df4-af06-635b51f05c8c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39112025-04-30T06:46:10.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
39122025-04-30T06:46:10.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ae97fae-a492-4231-a9cb-b29335497f05", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39132025-04-30T06:46:10.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
39142025-04-30T06:46:10.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a96440b-2ce9-44e7-ad21-4dd8b7087557", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39152025-04-30T06:46:10.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
39162025-04-30T06:46:10.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3cc6ca4-d11a-4113-b508-637ae7227357", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39172025-04-30T06:46:10.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
39182025-04-30T06:46:10.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2590c61e-ae4a-423d-b53a-8d36c3564336", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39192025-04-30T06:46:10.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
39202025-04-30T06:46:10.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a627d70d-9c3b-429b-9c55-dd128284d958", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39212025-04-30T06:46:10.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
39222025-04-30T06:46:10.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6136ed10-59de-47f0-b676-6963f44492e6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39232025-04-30T06:46:10.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
39242025-04-30T06:46:10.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a64e9dc-509a-4d1a-b77d-d0d669744963", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39252025-04-30T06:46:10.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
39262025-04-30T06:46:10.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1efc21-768b-4069-8c89-eae0eec3a27d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39272025-04-30T06:46:10.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
39282025-04-30T06:46:10.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94d86dc-3ffb-4a0c-abe5-1d294e1a30ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39292025-04-30T06:46:10.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
39302025-04-30T06:46:10.208ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39312025-04-30T06:46:10.208ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39322025-04-30T06:46:10.208ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39332025-04-30T06:46:10.208ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
39342025-04-30T06:46:10.210ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39352025-04-30T06:46:10.214ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39362025-04-30T06:46:10.224ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39372025-04-30T06:46:10.224ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39382025-04-30T06:46:10.224ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
39392025-04-30T06:46:10.225ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39402025-04-30T06:46:10.228ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39412025-04-30T06:46:10.228ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
39422025-04-30T06:46:10.230ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39432025-04-30T06:46:10.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d5c165e-62fa-4676-be70-276a748364ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39442025-04-30T06:46:10.235ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
39452025-04-30T06:46:10.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b01e827-3815-4116-bcd0-9679eabe8761", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39462025-04-30T06:46:10.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
39472025-04-30T06:46:10.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15014cd0-102f-4a5e-b454-dbb429d8963d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39482025-04-30T06:46:10.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
39492025-04-30T06:46:10.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a1d530-0dd3-4657-89c1-2ff9416d0bf1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39502025-04-30T06:46:10.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
39512025-04-30T06:46:10.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f26e2cc-dfa2-4758-bc25-9f044b69d306", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39522025-04-30T06:46:10.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
39532025-04-30T06:46:10.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d9bb1a-5114-4fd6-a094-e14126874bfa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39542025-04-30T06:46:10.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
39552025-04-30T06:46:10.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bc1db6f-55f5-4439-b6f5-2ebbd02b9aaf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39562025-04-30T06:46:10.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
39572025-04-30T06:46:10.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff57ee1-5fd9-41f0-a11c-8f4fc1045b20", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39582025-04-30T06:46:10.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
39592025-04-30T06:46:10.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8171bdc-0509-4a4b-b40e-58d8d55cc5fc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39602025-04-30T06:46:10.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
39612025-04-30T06:46:10.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d835b4c7-68f0-412f-b07a-f00117958a20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39622025-04-30T06:46:10.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
39632025-04-30T06:46:10.263ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
39642025-04-30T06:46:10.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04625caa-7f57-4f8b-8c8a-a935c444458d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39652025-04-30T06:46:10.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
39662025-04-30T06:46:10.269ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60f3030d-b46b-4f8f-abac-0bacd9b8c479", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39672025-04-30T06:46:10.269ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
39682025-04-30T06:46:10.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8463d7-90c5-403e-974f-025792c6b172", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39692025-04-30T06:46:10.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
39702025-04-30T06:46:10.275ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7f42679-9fb6-4dde-9820-0a30a2df226a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39712025-04-30T06:46:10.275ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
39722025-04-30T06:46:10.276ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "438b5059-45f5-480b-a129-be6755d78407", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39732025-04-30T06:46:10.276ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
39742025-04-30T06:46:10.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d2ce74-567e-4cc2-90bf-dcbe9e9dfc37", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39752025-04-30T06:46:10.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
39762025-04-30T06:46:10.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79fdb7d3-11a5-4138-b957-7250a8fdd105", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39772025-04-30T06:46:10.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
39782025-04-30T06:46:10.281ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93eeecaa-c25a-4e95-bf5f-08bac02c0143", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
39792025-04-30T06:46:10.281ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39802025-04-30T06:46:10.281ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39812025-04-30T06:46:10.281ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39822025-04-30T06:46:10.281ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39832025-04-30T06:46:10.284ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39842025-04-30T06:46:10.286ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39852025-04-30T06:46:10.294ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39862025-04-30T06:46:10.294ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39872025-04-30T06:46:10.294ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39882025-04-30T06:46:10.294ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39892025-04-30T06:46:10.294ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39902025-04-30T06:46:10.295ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39912025-04-30T06:46:10.304ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39922025-04-30T06:46:10.309ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39932025-04-30T06:46:10.309ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39942025-04-30T06:46:10.309ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39952025-04-30T06:46:10.310ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39962025-04-30T06:46:10.318ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39972025-04-30T06:46:10.318ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39982025-04-30T06:46:10.318ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39992025-04-30T06:46:10.318ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40002025-04-30T06:46:10.319ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40012025-04-30T06:46:10.323ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40022025-04-30T06:46:10.334ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40032025-04-30T06:46:10.334ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40042025-04-30T06:46:10.334ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40052025-04-30T06:46:10.335ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40062025-04-30T06:46:10.392ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
40072025-04-30T06:46:10.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
40082025-04-30T06:46:10.396ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:57867 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40092025-04-30T06:46:10.396ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40102025-04-30T06:46:10.396ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40112025-04-30T06:46:10.396ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40122025-04-30T06:46:10.397ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:57867 req_id = bcc4badc-19c7-4e1f-9822-4bd2aeb70bb5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40132025-04-30T06:46:10.397ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 604 local_addr = [::1]:54590 method = GET remote_addr = [::1]:57867 req_id = bcc4badc-19c7-4e1f-9822-4bd2aeb70bb5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40142025-04-30T06:46:10.398ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcc4badc-19c7-4e1f-9822-4bd2aeb70bb5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:09 GMT"} })
40152025-04-30T06:46:10.400ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40162025-04-30T06:46:10.401ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40172025-04-30T06:46:10.401ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
40182025-04-30T06:46:10.404ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40192025-04-30T06:46:10.404ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40202025-04-30T06:46:10.404ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40212025-04-30T06:46:10.404ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40222025-04-30T06:46:10.405ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40232025-04-30T06:46:10.408ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40242025-04-30T06:46:10.412ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40252025-04-30T06:46:10.413ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40262025-04-30T06:46:10.413ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40272025-04-30T06:46:10.416ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40282025-04-30T06:46:10.475ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
40292025-04-30T06:46:10.483ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40302025-04-30T06:46:10.483ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40312025-04-30T06:46:10.483ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 459.891374ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40322025-04-30T06:46:10.483ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40332025-04-30T06:46:10.484ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40342025-04-30T06:46:10.484ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40352025-04-30T06:46:10.484ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40362025-04-30T06:46:10.484ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40372025-04-30T06:46:10.486ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40382025-04-30T06:46:10.503ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40392025-04-30T06:46:10.508ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40402025-04-30T06:46:10.508ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40412025-04-30T06:46:10.508ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40422025-04-30T06:46:10.509ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40432025-04-30T06:46:10.512ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
40442025-04-30T06:46:10.513ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40452025-04-30T06:46:10.513ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cb af 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60608 req_id = a581bade-abd3-475d-b90a-a698a2212891
40462025-04-30T06:46:10.513ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52143,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60608 req_id = a581bade-abd3-475d-b90a-a698a2212891
40472025-04-30T06:46:10.514ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
40482025-04-30T06:46:10.514ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40492025-04-30T06:46:10.514ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60608 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a581bade-abd3-475d-b90a-a698a2212891
40502025-04-30T06:46:10.514ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4435, tv_nsec: 313490822 } })
40512025-04-30T06:46:10.514ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
40522025-04-30T06:46:10.515ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c4 3f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .?..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60008 req_id = 9257e1cc-1201-4be9-9019-95be00807394
40532025-04-30T06:46:10.515ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50239,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60008 req_id = 9257e1cc-1201-4be9-9019-95be00807394
40542025-04-30T06:46:10.515ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40552025-04-30T06:46:10.515ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60008 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9257e1cc-1201-4be9-9019-95be00807394
40562025-04-30T06:46:10.577ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40572025-04-30T06:46:10.578ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40582025-04-30T06:46:10.578ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40592025-04-30T06:46:10.578ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40602025-04-30T06:46:10.578ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40612025-04-30T06:46:10.579ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40622025-04-30T06:46:10.579ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b4 43 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .C..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60156 req_id = dce3a260-231e-461f-a192-423cba558ac2
40632025-04-30T06:46:10.579ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46147,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60156 req_id = dce3a260-231e-461f-a192-423cba558ac2
40642025-04-30T06:46:10.580ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
40652025-04-30T06:46:10.580ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46147, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60156 req_id = dce3a260-231e-461f-a192-423cba558ac2
40662025-04-30T06:46:10.580ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
40672025-04-30T06:46:10.580ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
40682025-04-30T06:46:10.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
40692025-04-30T06:46:10.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6355d5e-46d1-4f42-98fb-66429cc2fabc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40702025-04-30T06:46:10.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
40712025-04-30T06:46:10.585ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40722025-04-30T06:46:10.593ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40732025-04-30T06:46:10.593ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40742025-04-30T06:46:10.593ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40752025-04-30T06:46:10.593ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40762025-04-30T06:46:10.596ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40772025-04-30T06:46:10.596ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
40782025-04-30T06:46:10.596ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40792025-04-30T06:46:10.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7666f5-5784-436c-a6b4-fc48b3dd2685", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40802025-04-30T06:46:10.599ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
40812025-04-30T06:46:10.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f5816f8-93d8-4907-a6a9-ff9422256073", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40822025-04-30T06:46:10.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
40832025-04-30T06:46:10.605ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf377e3f-c533-467a-a59e-f5368d4af099", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40842025-04-30T06:46:10.606ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
40852025-04-30T06:46:10.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5822c85f-389b-4652-9cab-5ea8801108e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40862025-04-30T06:46:10.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
40872025-04-30T06:46:10.615ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f566ee6-7269-4642-8488-16bf5fa6175a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40882025-04-30T06:46:10.615ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
40892025-04-30T06:46:10.619ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68d018bd-faa2-4e55-bbdf-939f89575417", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40902025-04-30T06:46:10.619ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
40912025-04-30T06:46:10.625ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "880d0377-f429-42a8-b0f4-cee52e6ae8cc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40922025-04-30T06:46:10.625ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
40932025-04-30T06:46:10.628ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ce02e4-6a9e-46ab-8f8e-df54b816a3cd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40942025-04-30T06:46:10.628ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
40952025-04-30T06:46:10.628ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
40962025-04-30T06:46:10.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e260510-3b7f-4a9d-a169-710031609a38", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40972025-04-30T06:46:10.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
40982025-04-30T06:46:10.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c6c2d3-536c-4a12-a1e6-789259da71a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
40992025-04-30T06:46:10.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
41002025-04-30T06:46:10.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41012025-04-30T06:46:10.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41022025-04-30T06:46:10.638ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fcdd7c8-33da-465d-94b5-c72e02237a8e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41032025-04-30T06:46:10.638ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
41042025-04-30T06:46:10.641ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41052025-04-30T06:46:10.642ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41062025-04-30T06:46:10.643ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41072025-04-30T06:46:10.643ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41082025-04-30T06:46:10.643ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30598cb1-4a0b-4453-8a6c-3e4e00ca33b1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41092025-04-30T06:46:10.644ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
41102025-04-30T06:46:10.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71be98aa-13a0-49ab-a8ed-01fd75ad9d5a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41112025-04-30T06:46:10.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
41122025-04-30T06:46:10.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f55ef72-e202-4459-81bc-cb1cf5bc3502", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41132025-04-30T06:46:10.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
41142025-04-30T06:46:10.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebf5b2dc-1ad4-4b53-857a-86baaf9175a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41152025-04-30T06:46:10.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
41162025-04-30T06:46:10.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "957e3083-9012-4e1f-8687-4acbe702b1b1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41172025-04-30T06:46:10.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
41182025-04-30T06:46:10.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad8e96d-76eb-482c-9d6d-6c169a0936f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41192025-04-30T06:46:10.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
41202025-04-30T06:46:10.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1186472a-a400-4370-8136-55ff5758d8a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41212025-04-30T06:46:10.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
41222025-04-30T06:46:10.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b8eab92-be9c-4544-aff8-70f92a2ae170", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41232025-04-30T06:46:10.662ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
41242025-04-30T06:46:10.664ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc34598-4c30-444f-95b4-cf3a9aab811f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41252025-04-30T06:46:10.664ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
41262025-04-30T06:46:10.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff839ec4-84c4-4411-847d-a7c4d8b1f31e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41272025-04-30T06:46:10.667ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
41282025-04-30T06:46:10.668ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "490c73b3-4f09-4e6b-843d-f742dafc1ef9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41292025-04-30T06:46:10.668ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
41302025-04-30T06:46:10.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa8cf75e-eba5-4701-a12e-f14e2fa73459", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41312025-04-30T06:46:10.670ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
41322025-04-30T06:46:10.672ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "000a33a5-3002-4d78-9cee-9a681374a428", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41332025-04-30T06:46:10.672ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
41342025-04-30T06:46:10.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3071fc6-4540-4a78-92ab-82f9025ea755", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41352025-04-30T06:46:10.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
41362025-04-30T06:46:10.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "732f458f-28d8-4669-a396-46592168d3d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41372025-04-30T06:46:10.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
41382025-04-30T06:46:10.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c939feb-3d7e-4f74-abd6-b8b8fb52dd53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41392025-04-30T06:46:10.679ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
41402025-04-30T06:46:10.679ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41412025-04-30T06:46:10.679ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41422025-04-30T06:46:10.680ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41432025-04-30T06:46:10.680ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
41442025-04-30T06:46:10.680ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41452025-04-30T06:46:10.684ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41462025-04-30T06:46:10.691ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41472025-04-30T06:46:10.691ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41482025-04-30T06:46:10.691ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
41492025-04-30T06:46:10.692ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41502025-04-30T06:46:10.694ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41512025-04-30T06:46:10.694ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
41522025-04-30T06:46:10.695ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41532025-04-30T06:46:10.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5103004c-2645-4ffc-b303-436052ad87cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41542025-04-30T06:46:10.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
41552025-04-30T06:46:10.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61c0d4a2-32bf-4ce3-a27a-3d19d39b0f2e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41562025-04-30T06:46:10.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
41572025-04-30T06:46:10.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cc29fa6-7ac9-4e97-ad71-96d904880186", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41582025-04-30T06:46:10.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
41592025-04-30T06:46:10.710ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1965bec3-1a48-4613-9b0b-b82436fdf476", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41602025-04-30T06:46:10.710ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
41612025-04-30T06:46:10.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "164e1ed6-3cf1-4a5d-b518-61160cd70323", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41622025-04-30T06:46:10.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
41632025-04-30T06:46:10.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42acea3-8e6f-4bfd-b41c-19bd5f338ade", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41642025-04-30T06:46:10.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
41652025-04-30T06:46:10.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c067f3da-4aae-4d3c-8483-cde61f292ad2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41662025-04-30T06:46:10.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
41672025-04-30T06:46:10.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73ab81c-d1ee-4102-8a10-ad467f777931", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41682025-04-30T06:46:10.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
41692025-04-30T06:46:10.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7142dd36-a0f0-4883-9fc4-c281b1040e33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41702025-04-30T06:46:10.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
41712025-04-30T06:46:10.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bb90779-b836-45a0-9c36-f6947dc51056", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41722025-04-30T06:46:10.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
41732025-04-30T06:46:10.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bec2200f-32dc-476e-8cb6-34567ce9cdec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41742025-04-30T06:46:10.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
41752025-04-30T06:46:10.728ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
41762025-04-30T06:46:10.729ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9286aa0-dd57-49e2-9417-ccaa1084aa4b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41772025-04-30T06:46:10.730ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
41782025-04-30T06:46:10.731ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c04b94-611c-48a9-ba37-1f5a6d4b4991", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41792025-04-30T06:46:10.731ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
41802025-04-30T06:46:10.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4c1da0b-fb73-4c0e-8034-ebd5050a7b85", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41812025-04-30T06:46:10.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
41822025-04-30T06:46:10.733ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39c4e2c9-9541-4d42-b345-e82c27325b91", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41832025-04-30T06:46:10.733ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
41842025-04-30T06:46:10.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "935e2d4c-fafd-4a44-a0ea-b20a4bc6ae8e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41852025-04-30T06:46:10.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
41862025-04-30T06:46:10.738ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d840f814-2917-48ad-9d0a-068209905fe1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
41872025-04-30T06:46:10.738ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
41882025-04-30T06:46:10.739ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41892025-04-30T06:46:10.739ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41902025-04-30T06:46:10.739ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41912025-04-30T06:46:10.742ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41922025-04-30T06:46:10.745ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41932025-04-30T06:46:10.751ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
41942025-04-30T06:46:10.760ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
41952025-04-30T06:46:10.760ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41962025-04-30T06:46:10.760ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41972025-04-30T06:46:10.760ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41982025-04-30T06:46:10.761ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41992025-04-30T06:46:10.765ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42002025-04-30T06:46:10.772ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42012025-04-30T06:46:10.772ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42022025-04-30T06:46:10.772ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42032025-04-30T06:46:10.774ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42042025-04-30T06:46:10.851ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
42052025-04-30T06:46:10.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
42062025-04-30T06:46:10.851ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42072025-04-30T06:46:10.851ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42082025-04-30T06:46:10.851ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42092025-04-30T06:46:10.851ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42102025-04-30T06:46:10.852ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42112025-04-30T06:46:10.852ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:42913 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42122025-04-30T06:46:10.854ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:42913 req_id = 5a73c4e1-b5bb-4f89-99bc-62d3ee652aa2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42132025-04-30T06:46:10.854ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 567 local_addr = [::1]:54590 method = GET remote_addr = [::1]:42913 req_id = 5a73c4e1-b5bb-4f89-99bc-62d3ee652aa2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42142025-04-30T06:46:10.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a73c4e1-b5bb-4f89-99bc-62d3ee652aa2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
42152025-04-30T06:46:10.860ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42162025-04-30T06:46:10.860ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42172025-04-30T06:46:10.860ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
42182025-04-30T06:46:10.863ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42192025-04-30T06:46:10.863ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42202025-04-30T06:46:10.863ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42212025-04-30T06:46:10.863ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42222025-04-30T06:46:10.864ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42232025-04-30T06:46:10.865ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42242025-04-30T06:46:10.871ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42252025-04-30T06:46:10.871ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42262025-04-30T06:46:10.871ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42272025-04-30T06:46:10.871ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42282025-04-30T06:46:10.873ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42292025-04-30T06:46:10.884ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42302025-04-30T06:46:10.885ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42312025-04-30T06:46:10.885ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42322025-04-30T06:46:10.885ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42332025-04-30T06:46:10.889ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42342025-04-30T06:46:10.889ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42352025-04-30T06:46:10.889ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42362025-04-30T06:46:10.943ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42372025-04-30T06:46:10.944ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42382025-04-30T06:46:10.944ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 460.353454ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42392025-04-30T06:46:10.944ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42402025-04-30T06:46:10.944ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42412025-04-30T06:46:10.944ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42422025-04-30T06:46:10.944ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42432025-04-30T06:46:10.944ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42442025-04-30T06:46:10.945ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42452025-04-30T06:46:10.949ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
42462025-04-30T06:46:10.949ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42472025-04-30T06:46:10.954ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42482025-04-30T06:46:10.954ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42492025-04-30T06:46:10.954ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42502025-04-30T06:46:10.954ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42512025-04-30T06:46:10.958ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
42522025-04-30T06:46:10.958ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42532025-04-30T06:46:10.959ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6e 44 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 nD..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51417 req_id = 227d403a-227b-46c2-8279-e2c8412a39ba
42542025-04-30T06:46:10.959ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28228,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51417 req_id = 227d403a-227b-46c2-8279-e2c8412a39ba
42552025-04-30T06:46:10.959ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
42562025-04-30T06:46:10.959ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42572025-04-30T06:46:10.960ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51417 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 227d403a-227b-46c2-8279-e2c8412a39ba
42582025-04-30T06:46:10.961ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4435, tv_nsec: 759932719 } })
42592025-04-30T06:46:10.961ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
42602025-04-30T06:46:10.961ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 37 30 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 70..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54155 req_id = cf5fb596-6df0-41f3-93b3-b079f1a59202
42612025-04-30T06:46:10.961ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14128,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54155 req_id = cf5fb596-6df0-41f3-93b3-b079f1a59202
42622025-04-30T06:46:10.962ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42632025-04-30T06:46:10.962ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54155 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cf5fb596-6df0-41f3-93b3-b079f1a59202
42642025-04-30T06:46:11.040ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42652025-04-30T06:46:11.041ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42662025-04-30T06:46:11.041ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42672025-04-30T06:46:11.041ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42682025-04-30T06:46:11.041ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42692025-04-30T06:46:11.042ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42702025-04-30T06:46:11.042ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42712025-04-30T06:46:11.042ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42722025-04-30T06:46:11.044ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
42732025-04-30T06:46:11.045ZDEBGtest_multiple_snapshots (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
42742025-04-30T06:46:11.045ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
42752025-04-30T06:46:11.045ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: eb 6e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .n..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64768 req_id = 5c1a9de2-5ad1-4f89-836d-d8a4020bb3f1
42762025-04-30T06:46:11.045ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60270,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64768 req_id = 5c1a9de2-5ad1-4f89-836d-d8a4020bb3f1
42772025-04-30T06:46:11.046ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
42782025-04-30T06:46:11.046ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64768 req_id = 5c1a9de2-5ad1-4f89-836d-d8a4020bb3f1
42792025-04-30T06:46:11.046ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
42802025-04-30T06:46:11.046ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
42812025-04-30T06:46:11.046ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
42822025-04-30T06:46:11.046ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42832025-04-30T06:46:11.055ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42842025-04-30T06:46:11.055ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42852025-04-30T06:46:11.055ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42862025-04-30T06:46:11.055ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42872025-04-30T06:46:11.068ZTRACtest_multiple_snapshots (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
42882025-04-30T06:46:11.069ZDEBGtest_multiple_snapshots (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:46:11.045366813Z } timeseries_name = integration_target:integration_metric
42892025-04-30T06:46:11.069ZDEBGtest_multiple_snapshots (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
42902025-04-30T06:46:11.069ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
42912025-04-30T06:46:11.077ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42922025-04-30T06:46:11.078ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
42932025-04-30T06:46:11.078ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42942025-04-30T06:46:11.079ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42952025-04-30T06:46:11.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "25add179-7d08-4e00-b911-1e64fc2f666a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
42962025-04-30T06:46:11.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
42972025-04-30T06:46:11.084ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
42982025-04-30T06:46:11.084ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42992025-04-30T06:46:11.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "022a7f8a-cc2c-4ca9-84d5-8982e87c42a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43002025-04-30T06:46:11.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
43012025-04-30T06:46:11.087ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
43022025-04-30T06:46:11.087ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43032025-04-30T06:46:11.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f528f3c1-3c77-49a4-ab6f-de48f2f8b9b4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43042025-04-30T06:46:11.098ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
43052025-04-30T06:46:11.099ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77f9b04d-9e5d-4f88-862c-296d63c53690", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43062025-04-30T06:46:11.099ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
43072025-04-30T06:46:11.101ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86d5d4b6-7ac3-4eaf-8af5-9a17b738bf37", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43082025-04-30T06:46:11.101ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
43092025-04-30T06:46:11.106ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aea2cfa-a04e-4f11-b60e-9821b8ef7d6e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43102025-04-30T06:46:11.106ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
43112025-04-30T06:46:11.111ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eaa109d-a331-469a-b8ec-f7b6e2f26aa9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43122025-04-30T06:46:11.112ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
43132025-04-30T06:46:11.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77755b57-1abe-48c9-8c4f-36360044fd96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43142025-04-30T06:46:11.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
43152025-04-30T06:46:11.122ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
43162025-04-30T06:46:11.125ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41af1a78-6984-4f70-aa8a-0e323c48802e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43172025-04-30T06:46:11.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
43182025-04-30T06:46:11.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2a1334e-84cd-40cc-ace4-eeed82e500b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43192025-04-30T06:46:11.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
43202025-04-30T06:46:11.132ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8737dd1-4d75-4044-9197-ced1b03c5f94", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43212025-04-30T06:46:11.133ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
43222025-04-30T06:46:11.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4928ee91-545a-473d-a5fd-3fdf66fc77ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43232025-04-30T06:46:11.135ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
43242025-04-30T06:46:11.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d94796-4dac-4232-9a27-963c92f58ed1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43252025-04-30T06:46:11.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
43262025-04-30T06:46:11.139ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26515e84-db72-4af7-a7ca-40a6f2ea4a54", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43272025-04-30T06:46:11.139ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
43282025-04-30T06:46:11.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17db312-2d7b-4fc1-88e8-3369a4fd385a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43292025-04-30T06:46:11.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
43302025-04-30T06:46:11.154ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
43312025-04-30T06:46:11.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f46bb28a-b227-42ef-a14c-51d80e48484e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43322025-04-30T06:46:11.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
43332025-04-30T06:46:11.164ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c807a6-5604-430d-993b-642f803699c5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43342025-04-30T06:46:11.164ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
43352025-04-30T06:46:11.168ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6385ecf-c4e0-49d3-8fa2-26195eaf2e54", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43362025-04-30T06:46:11.168ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
43372025-04-30T06:46:11.169ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62c489b3-56fb-4872-bed7-2f60c346f33a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43382025-04-30T06:46:11.169ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
43392025-04-30T06:46:11.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d64c04a4-af31-42cd-9443-ea76d620c29f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43402025-04-30T06:46:11.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
43412025-04-30T06:46:11.173ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43422025-04-30T06:46:11.173ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43432025-04-30T06:46:11.173ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43442025-04-30T06:46:11.173ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
43452025-04-30T06:46:11.174ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43462025-04-30T06:46:11.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ff08a01-7b80-4f3f-b4e2-256e6b3917eb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43472025-04-30T06:46:11.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
43482025-04-30T06:46:11.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa9ee3d-b369-42ca-b10d-219e3469fb31", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43492025-04-30T06:46:11.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
43502025-04-30T06:46:11.177ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43512025-04-30T06:46:11.183ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43522025-04-30T06:46:11.183ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43532025-04-30T06:46:11.183ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
43542025-04-30T06:46:11.183ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43552025-04-30T06:46:11.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5808c24c-12ef-4965-a8ab-0e1c105f40bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43562025-04-30T06:46:11.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
43572025-04-30T06:46:11.186ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43582025-04-30T06:46:11.186ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
43592025-04-30T06:46:11.187ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43602025-04-30T06:46:11.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e275e72-7295-4924-9521-5910dd59f529", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43612025-04-30T06:46:11.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
43622025-04-30T06:46:11.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff7ba158-633a-4a6e-a9f5-bcf98e658806", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43632025-04-30T06:46:11.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
43642025-04-30T06:46:11.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e836622-c262-4db8-9645-20a7f7c41104", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43652025-04-30T06:46:11.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
43662025-04-30T06:46:11.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e2c27d-5684-47e5-862b-157a3b9638e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43672025-04-30T06:46:11.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
43682025-04-30T06:46:11.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53b36f4b-7c57-46f3-8b92-c346390992fe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43692025-04-30T06:46:11.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
43702025-04-30T06:46:11.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d25457c5-f02f-4995-827f-8c62d1e2fd0d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43712025-04-30T06:46:11.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
43722025-04-30T06:46:11.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "549915bd-00f9-415c-8b4b-2e60738b8b75", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43732025-04-30T06:46:11.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
43742025-04-30T06:46:11.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7402a412-b511-4ea9-978f-4453c9e3a833", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43752025-04-30T06:46:11.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
43762025-04-30T06:46:11.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a6f75f-3110-4656-b033-09680db8f9b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43772025-04-30T06:46:11.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
43782025-04-30T06:46:11.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c449911f-5f48-4e27-8787-68442117b22a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43792025-04-30T06:46:11.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
43802025-04-30T06:46:11.226ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d71d20e7-d8f3-49e1-9da2-15afc2eaf90d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43812025-04-30T06:46:11.226ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
43822025-04-30T06:46:11.229ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd460a4f-559e-4548-b3de-33f637232f89", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43832025-04-30T06:46:11.229ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
43842025-04-30T06:46:11.233ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
43852025-04-30T06:46:11.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73fa8db5-84eb-49d1-9cfa-13e3fb9df2ab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43862025-04-30T06:46:11.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
43872025-04-30T06:46:11.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca89c752-f069-4ad9-b088-27c7b3eb5fea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43882025-04-30T06:46:11.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
43892025-04-30T06:46:11.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5295db84-7738-4680-aa74-7fbd0d279530", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43902025-04-30T06:46:11.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
43912025-04-30T06:46:11.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac8e7e1-7188-40f7-9b68-72fa15a748ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43922025-04-30T06:46:11.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
43932025-04-30T06:46:11.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b493f78-b40f-4632-82d7-9c064bc46a5f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43942025-04-30T06:46:11.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
43952025-04-30T06:46:11.251ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb914b2b-d07b-47c0-9b4a-d2e16555337c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43962025-04-30T06:46:11.251ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
43972025-04-30T06:46:11.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ff1e06-44f0-4252-ba83-c2b5f775ea38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
43982025-04-30T06:46:11.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
43992025-04-30T06:46:11.255ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "648911a3-503a-45f6-96fe-115e5071b401", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
44002025-04-30T06:46:11.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
44012025-04-30T06:46:11.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3393081a-30fa-48d3-a386-9a3932ce4fe8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
44022025-04-30T06:46:11.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
44032025-04-30T06:46:11.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f3d19b-f1c6-496c-a734-66679cd1761a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
44042025-04-30T06:46:11.261ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44052025-04-30T06:46:11.261ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44062025-04-30T06:46:11.261ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44072025-04-30T06:46:11.261ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44082025-04-30T06:46:11.262ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44092025-04-30T06:46:11.266ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44102025-04-30T06:46:11.270ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44112025-04-30T06:46:11.270ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44122025-04-30T06:46:11.270ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44132025-04-30T06:46:11.270ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44142025-04-30T06:46:11.270ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44152025-04-30T06:46:11.271ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44162025-04-30T06:46:11.285ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44172025-04-30T06:46:11.285ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44182025-04-30T06:46:11.285ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44192025-04-30T06:46:11.285ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
44202025-04-30T06:46:11.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44212025-04-30T06:46:11.288ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44222025-04-30T06:46:11.292ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44232025-04-30T06:46:11.293ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44242025-04-30T06:46:11.293ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44252025-04-30T06:46:11.293ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44262025-04-30T06:46:11.295ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44272025-04-30T06:46:11.304ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44282025-04-30T06:46:11.304ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44292025-04-30T06:46:11.304ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
44302025-04-30T06:46:11.304ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44312025-04-30T06:46:11.383ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
44322025-04-30T06:46:11.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
44332025-04-30T06:46:11.384ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:39427 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44342025-04-30T06:46:11.385ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:39427 req_id = 1da1b944-660a-476a-8fbc-f66c9bb2ee4f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44352025-04-30T06:46:11.386ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 491 local_addr = [::1]:54590 method = GET remote_addr = [::1]:39427 req_id = 1da1b944-660a-476a-8fbc-f66c9bb2ee4f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44362025-04-30T06:46:11.386ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44372025-04-30T06:46:11.386ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
44382025-04-30T06:46:11.386ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44392025-04-30T06:46:11.387ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1da1b944-660a-476a-8fbc-f66c9bb2ee4f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
44402025-04-30T06:46:11.391ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44412025-04-30T06:46:11.391ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44422025-04-30T06:46:11.391ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
44432025-04-30T06:46:11.394ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44442025-04-30T06:46:11.394ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44452025-04-30T06:46:11.394ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44462025-04-30T06:46:11.395ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44472025-04-30T06:46:11.395ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44482025-04-30T06:46:11.399ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44492025-04-30T06:46:11.405ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44502025-04-30T06:46:11.405ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44512025-04-30T06:46:11.405ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44522025-04-30T06:46:11.406ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44532025-04-30T06:46:11.461ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44542025-04-30T06:46:11.462ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44552025-04-30T06:46:11.462ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 517.886476ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44562025-04-30T06:46:11.462ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44572025-04-30T06:46:11.462ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44582025-04-30T06:46:11.462ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44592025-04-30T06:46:11.462ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44602025-04-30T06:46:11.462ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44612025-04-30T06:46:11.463ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44622025-04-30T06:46:11.464ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
44632025-04-30T06:46:11.465ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44642025-04-30T06:46:11.470ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44652025-04-30T06:46:11.470ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44662025-04-30T06:46:11.470ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44672025-04-30T06:46:11.471ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44682025-04-30T06:46:11.479ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
44692025-04-30T06:46:11.479ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44702025-04-30T06:46:11.483ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 49 f9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 I............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61473 req_id = 53b84be7-7905-4190-a219-eb1e274d2c0f
44712025-04-30T06:46:11.483ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18937,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61473 req_id = 53b84be7-7905-4190-a219-eb1e274d2c0f
44722025-04-30T06:46:11.484ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
44732025-04-30T06:46:11.484ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44742025-04-30T06:46:11.484ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61473 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 53b84be7-7905-4190-a219-eb1e274d2c0f
44752025-04-30T06:46:11.485ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4436, tv_nsec: 284237362 } })
44762025-04-30T06:46:11.485ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
44772025-04-30T06:46:11.486ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 96 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54362 req_id = 4789f1a9-3869-4e73-9184-92e65add064a
44782025-04-30T06:46:11.486ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33942,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54362 req_id = 4789f1a9-3869-4e73-9184-92e65add064a
44792025-04-30T06:46:11.486ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44802025-04-30T06:46:11.486ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54362 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4789f1a9-3869-4e73-9184-92e65add064a
44812025-04-30T06:46:11.529ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
44822025-04-30T06:46:11.529ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44832025-04-30T06:46:11.529ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44842025-04-30T06:46:11.529ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44852025-04-30T06:46:11.529ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
44862025-04-30T06:46:11.530ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44872025-04-30T06:46:11.530ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0c 4a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .J..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62807 req_id = 2f339926-647c-43db-bc35-4864cba0c7a2
44882025-04-30T06:46:11.531ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3146,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62807 req_id = 2f339926-647c-43db-bc35-4864cba0c7a2
44892025-04-30T06:46:11.531ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
44902025-04-30T06:46:11.531ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3146, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62807 req_id = 2f339926-647c-43db-bc35-4864cba0c7a2
44912025-04-30T06:46:11.531ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
44922025-04-30T06:46:11.531ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
44932025-04-30T06:46:11.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
44942025-04-30T06:46:11.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a092707-a921-4670-9f35-151a368dbc06", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
44952025-04-30T06:46:11.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
44962025-04-30T06:46:11.533ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44972025-04-30T06:46:11.542ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44982025-04-30T06:46:11.542ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44992025-04-30T06:46:11.542ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
45002025-04-30T06:46:11.543ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45012025-04-30T06:46:11.546ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45022025-04-30T06:46:11.546ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
45032025-04-30T06:46:11.546ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45042025-04-30T06:46:11.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2242e6d5-650e-496b-afa6-97f20d6bf633", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45052025-04-30T06:46:11.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
45062025-04-30T06:46:11.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
45072025-04-30T06:46:11.548ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45082025-04-30T06:46:11.549ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 3b14a024-4767-4b9a-86cb-b9a96586c260 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45092025-04-30T06:46:11.549ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 3b14a024-4767-4b9a-86cb-b9a96586c260 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45102025-04-30T06:46:11.549ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 3b14a024-4767-4b9a-86cb-b9a96586c260 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45112025-04-30T06:46:11.551ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 3b14a024-4767-4b9a-86cb-b9a96586c260 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45122025-04-30T06:46:11.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c957c7f-b649-410e-96c4-514213a7a140", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45132025-04-30T06:46:11.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
45142025-04-30T06:46:11.554ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 4819 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 3b14a024-4767-4b9a-86cb-b9a96586c260 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45152025-04-30T06:46:11.554ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b14a024-4767-4b9a-86cb-b9a96586c260", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45162025-04-30T06:46:11.555ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45172025-04-30T06:46:11.555ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 027cedb9-74b8-4407-bf9c-f0809e28b232 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45182025-04-30T06:46:11.555ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 027cedb9-74b8-4407-bf9c-f0809e28b232 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45192025-04-30T06:46:11.555ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 027cedb9-74b8-4407-bf9c-f0809e28b232 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45202025-04-30T06:46:11.556ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 027cedb9-74b8-4407-bf9c-f0809e28b232 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45212025-04-30T06:46:11.557ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2235377f-80ad-40df-8f82-0a70c38e9702", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45222025-04-30T06:46:11.557ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
45232025-04-30T06:46:11.559ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 3791 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 027cedb9-74b8-4407-bf9c-f0809e28b232 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45242025-04-30T06:46:11.561ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "027cedb9-74b8-4407-bf9c-f0809e28b232", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45252025-04-30T06:46:11.561ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
45262025-04-30T06:46:11.561ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45272025-04-30T06:46:11.561ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
45282025-04-30T06:46:11.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65a4b6f4-3e64-4c81-a45b-6e14f2432909", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45292025-04-30T06:46:11.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
45302025-04-30T06:46:11.566ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13dc31f7-c709-44df-a9e5-12255f7ae976", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45312025-04-30T06:46:11.566ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
45322025-04-30T06:46:11.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06162c9b-f20f-4626-9019-aa4135a17442", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45332025-04-30T06:46:11.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
45342025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fef2611-9068-4561-8740-56800f73faa7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45352025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
45362025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4a4046-1d16-4ab5-8580-316865624be6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45372025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
45382025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b31d70e0-8495-4b49-bfc4-e78227aeac05", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45392025-04-30T06:46:11.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
45402025-04-30T06:46:11.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55689f9a-d52f-417a-bab0-4627d9345901", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45412025-04-30T06:46:11.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
45422025-04-30T06:46:11.578ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
45432025-04-30T06:46:11.579ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2b21215-4266-42ce-a18c-13fa4af09303", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:10 GMT"} })
45442025-04-30T06:46:11.579ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
45452025-04-30T06:46:11.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a75f9fd0-7dcb-4fb6-b5f5-9372cfcc242c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45462025-04-30T06:46:11.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
45472025-04-30T06:46:11.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c547b465-bf39-4d1f-9302-c65c7f5af5bb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45482025-04-30T06:46:11.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
45492025-04-30T06:46:11.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10057527-db51-4ff0-828f-6d8465981d7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45502025-04-30T06:46:11.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
45512025-04-30T06:46:11.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9824cc5d-8446-4592-a0c7-0dd6dbf0e566", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45522025-04-30T06:46:11.595ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
45532025-04-30T06:46:11.599ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d090b59d-f6a7-4ac2-a8e9-b6683c2d4b42", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45542025-04-30T06:46:11.600ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
45552025-04-30T06:46:11.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a938a3-affe-4332-9923-a0f1fd6b890b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45562025-04-30T06:46:11.609ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
45572025-04-30T06:46:11.612ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35c6920-4d5e-4085-aae0-acae8a0e8762", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45582025-04-30T06:46:11.612ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
45592025-04-30T06:46:11.615ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7715d8a9-c95b-4d0d-9ef3-8b1c067eaa76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45602025-04-30T06:46:11.616ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
45612025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0b2ecc9-9a4d-4e91-a966-ffa54a4740b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45622025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
45632025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99557102-fa0e-46cb-9cdb-49b2ce4ef0ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45642025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
45652025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c654ce70-8232-4dce-b087-098d5b0b68b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45662025-04-30T06:46:11.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
45672025-04-30T06:46:11.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5df366-78ee-4824-b4ce-183f042ed6e4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45682025-04-30T06:46:11.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
45692025-04-30T06:46:11.625ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a50871c6-d145-475e-a78b-107ba0785d16", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45702025-04-30T06:46:11.625ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
45712025-04-30T06:46:11.627ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4251e3ec-1198-4881-877b-e8513c726913", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45722025-04-30T06:46:11.627ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
45732025-04-30T06:46:11.628ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45742025-04-30T06:46:11.628ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45752025-04-30T06:46:11.628ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45762025-04-30T06:46:11.628ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
45772025-04-30T06:46:11.630ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45782025-04-30T06:46:11.633ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45792025-04-30T06:46:11.639ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45802025-04-30T06:46:11.639ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45812025-04-30T06:46:11.639ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
45822025-04-30T06:46:11.640ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45832025-04-30T06:46:11.642ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45842025-04-30T06:46:11.642ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45852025-04-30T06:46:11.644ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45862025-04-30T06:46:11.644ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
45872025-04-30T06:46:11.645ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45882025-04-30T06:46:11.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da3e1475-21e8-4d76-8c15-681076c4e7ee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45892025-04-30T06:46:11.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
45902025-04-30T06:46:11.646ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45912025-04-30T06:46:11.647ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45922025-04-30T06:46:11.647ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45932025-04-30T06:46:11.647ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45942025-04-30T06:46:11.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f33267d2-7b99-4278-90b4-9c3944bf5ae9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45952025-04-30T06:46:11.651ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
45962025-04-30T06:46:11.653ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "072be0d4-0314-45b1-8cdd-6bab6f14cc85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45972025-04-30T06:46:11.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
45982025-04-30T06:46:11.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ece1b51d-53f1-48b4-83e0-3e3e809f4134", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
45992025-04-30T06:46:11.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
46002025-04-30T06:46:11.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6c14456-b09b-47da-b005-454dca45d843", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46012025-04-30T06:46:11.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
46022025-04-30T06:46:11.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f33e1f1-bcda-436d-b2ce-c9f50229b0e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46032025-04-30T06:46:11.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
46042025-04-30T06:46:11.662ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fca5b90-8172-4754-9792-3108afd968a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46052025-04-30T06:46:11.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
46062025-04-30T06:46:11.664ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff419c5-4692-460a-a3bf-a2c9d338cdd7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46072025-04-30T06:46:11.665ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
46082025-04-30T06:46:11.672ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d9c1ef1-e211-45e8-a44f-e81af2ad5aaa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46092025-04-30T06:46:11.673ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
46102025-04-30T06:46:11.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b108fbb0-2a65-4d38-91fe-05dc21da5500", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46112025-04-30T06:46:11.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
46122025-04-30T06:46:11.682ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
46132025-04-30T06:46:11.685ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1de5ef0-f625-41ce-a344-bd857458b47b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46142025-04-30T06:46:11.685ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
46152025-04-30T06:46:11.689ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fa11aa6-19c0-47a6-8b33-2f6288e601f0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46162025-04-30T06:46:11.689ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
46172025-04-30T06:46:11.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "595f8bd7-97e7-47fd-b41e-53014c9d2b9d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46182025-04-30T06:46:11.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
46192025-04-30T06:46:11.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d41e71-35e5-4fb7-b3dc-d65a2c013db8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46202025-04-30T06:46:11.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
46212025-04-30T06:46:11.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f3b6b84-78ae-4ff8-b117-e070b31fe1ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46222025-04-30T06:46:11.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
46232025-04-30T06:46:11.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef6b5def-1f6a-48f6-9698-979fca0775a5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46242025-04-30T06:46:11.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
46252025-04-30T06:46:11.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "495cc215-0ada-4218-b2ae-8000e5ed4a39", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46262025-04-30T06:46:11.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
46272025-04-30T06:46:11.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e54705-e045-4591-8e1a-2e3942c29ef7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46282025-04-30T06:46:11.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
46292025-04-30T06:46:11.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "374eeaaf-f997-46d8-ac4a-0b0f0e83e807", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46302025-04-30T06:46:11.704ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46312025-04-30T06:46:11.704ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46322025-04-30T06:46:11.704ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46332025-04-30T06:46:11.704ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46342025-04-30T06:46:11.706ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46352025-04-30T06:46:11.710ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46362025-04-30T06:46:11.717ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46372025-04-30T06:46:11.717ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46382025-04-30T06:46:11.718ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46392025-04-30T06:46:11.718ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46402025-04-30T06:46:11.718ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46412025-04-30T06:46:11.720ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46422025-04-30T06:46:11.722ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46432025-04-30T06:46:11.730ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46442025-04-30T06:46:11.730ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46452025-04-30T06:46:11.730ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46462025-04-30T06:46:11.731ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46472025-04-30T06:46:11.734ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46482025-04-30T06:46:11.734ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46492025-04-30T06:46:11.734ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46502025-04-30T06:46:11.734ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
46512025-04-30T06:46:11.735ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46522025-04-30T06:46:11.851ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
46532025-04-30T06:46:11.863ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
46542025-04-30T06:46:11.863ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:60062 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46552025-04-30T06:46:11.865ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:60062 req_id = 493cc308-8fbc-4f7c-b9f8-c46293376776 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46562025-04-30T06:46:11.865ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 470 local_addr = [::1]:54590 method = GET remote_addr = [::1]:60062 req_id = 493cc308-8fbc-4f7c-b9f8-c46293376776 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46572025-04-30T06:46:11.865ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "493cc308-8fbc-4f7c-b9f8-c46293376776", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
46582025-04-30T06:46:11.867ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46592025-04-30T06:46:11.867ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46602025-04-30T06:46:11.867ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
46612025-04-30T06:46:11.870ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46622025-04-30T06:46:11.870ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46632025-04-30T06:46:11.871ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46642025-04-30T06:46:11.871ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46652025-04-30T06:46:11.871ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46662025-04-30T06:46:11.872ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46672025-04-30T06:46:11.890ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46682025-04-30T06:46:11.890ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46692025-04-30T06:46:11.890ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
46702025-04-30T06:46:11.893ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46712025-04-30T06:46:11.894ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46722025-04-30T06:46:11.901ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46732025-04-30T06:46:11.902ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46742025-04-30T06:46:11.902ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46752025-04-30T06:46:11.903ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46762025-04-30T06:46:11.904ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46772025-04-30T06:46:11.905ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
46782025-04-30T06:46:11.905ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46792025-04-30T06:46:11.966ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
46802025-04-30T06:46:11.967ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
46812025-04-30T06:46:11.967ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 504.939659ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
46822025-04-30T06:46:11.967ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
46832025-04-30T06:46:11.967ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46842025-04-30T06:46:11.967ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46852025-04-30T06:46:11.967ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46862025-04-30T06:46:11.967ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46872025-04-30T06:46:11.968ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46882025-04-30T06:46:11.978ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
46892025-04-30T06:46:11.979ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46902025-04-30T06:46:11.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46912025-04-30T06:46:11.986ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46922025-04-30T06:46:11.986ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46932025-04-30T06:46:11.988ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46942025-04-30T06:46:11.990ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46952025-04-30T06:46:11.990ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46962025-04-30T06:46:11.991ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
46972025-04-30T06:46:11.992ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46982025-04-30T06:46:11.993ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
46992025-04-30T06:46:11.993ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47002025-04-30T06:46:11.995ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c9 7f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58810 req_id = 78b92f06-e2f3-43d7-8d7c-57401d946d25
47012025-04-30T06:46:11.995ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51583,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58810 req_id = 78b92f06-e2f3-43d7-8d7c-57401d946d25
47022025-04-30T06:46:11.995ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
47032025-04-30T06:46:11.996ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47042025-04-30T06:46:11.996ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58810 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 78b92f06-e2f3-43d7-8d7c-57401d946d25
47052025-04-30T06:46:11.997ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
47062025-04-30T06:46:11.997ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47072025-04-30T06:46:11.998ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4436, tv_nsec: 797217962 } })
47082025-04-30T06:46:11.998ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
47092025-04-30T06:46:11.999ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 83 bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58899 req_id = a03f3d86-9d39-4040-bb28-5f2ef58c38b6
47102025-04-30T06:46:11.999ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33724,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58899 req_id = a03f3d86-9d39-4040-bb28-5f2ef58c38b6
47112025-04-30T06:46:12.003ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47122025-04-30T06:46:12.004ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58899 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a03f3d86-9d39-4040-bb28-5f2ef58c38b6
47132025-04-30T06:46:12.004ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
47142025-04-30T06:46:12.005ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47152025-04-30T06:46:12.145ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47162025-04-30T06:46:12.146ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47172025-04-30T06:46:12.146ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47182025-04-30T06:46:12.146ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47192025-04-30T06:46:12.146ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47202025-04-30T06:46:12.147ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47212025-04-30T06:46:12.147ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47222025-04-30T06:46:12.148ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 50 b9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c P............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57851 req_id = c538b30f-f0a4-4c43-a0d8-354251f9aae8
47232025-04-30T06:46:12.148ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20665,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57851 req_id = c538b30f-f0a4-4c43-a0d8-354251f9aae8
47242025-04-30T06:46:12.148ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
47252025-04-30T06:46:12.148ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20665, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57851 req_id = c538b30f-f0a4-4c43-a0d8-354251f9aae8
47262025-04-30T06:46:12.149ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
47272025-04-30T06:46:12.149ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
47282025-04-30T06:46:12.149ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
47292025-04-30T06:46:12.150ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47302025-04-30T06:46:12.168ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47312025-04-30T06:46:12.169ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47322025-04-30T06:46:12.174ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47332025-04-30T06:46:12.186ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47342025-04-30T06:46:12.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "36285c0d-8afa-44fe-92bf-713daa947355", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47352025-04-30T06:46:12.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
47362025-04-30T06:46:12.203ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47372025-04-30T06:46:12.203ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47382025-04-30T06:46:12.206ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47392025-04-30T06:46:12.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c5f47d5-4a37-4325-a8a5-5c6fa67b94a8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47402025-04-30T06:46:12.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
47412025-04-30T06:46:12.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "feb69797-0730-4539-ad7b-eafeb2074ab1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47422025-04-30T06:46:12.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
47432025-04-30T06:46:12.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d10281ee-d689-4a67-a998-7601357ad1a5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47442025-04-30T06:46:12.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
47452025-04-30T06:46:12.224ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "909526b8-2f35-4d32-af68-98b8148f98a7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47462025-04-30T06:46:12.224ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
47472025-04-30T06:46:12.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e8bdd2e-c5f9-439a-8109-cf578b83356d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47482025-04-30T06:46:12.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
47492025-04-30T06:46:12.235ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1afc7e6-8b12-401c-82fa-d5acd55b993b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47502025-04-30T06:46:12.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
47512025-04-30T06:46:12.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ff451bb-089b-4775-a9d5-583fef224fb7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47522025-04-30T06:46:12.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
47532025-04-30T06:46:12.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e9c8add-f894-4bbb-9628-402dfa2865d6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47542025-04-30T06:46:12.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
47552025-04-30T06:46:12.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd984cc-e224-4daa-9991-0abb8b9478ed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47562025-04-30T06:46:12.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
47572025-04-30T06:46:12.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f82c2bd2-a8ca-428d-906a-4c9b52f2114a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47582025-04-30T06:46:12.252ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
47592025-04-30T06:46:12.253ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
47602025-04-30T06:46:12.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e878fc17-b71e-461a-9f3d-d6aa076d507c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47612025-04-30T06:46:12.281ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
47622025-04-30T06:46:12.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88fb919f-2904-4cf9-b202-30aa9331ebab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47632025-04-30T06:46:12.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
47642025-04-30T06:46:12.286ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b61bcac9-74ab-4dec-84d4-3d3d362d67c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47652025-04-30T06:46:12.286ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
47662025-04-30T06:46:12.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b9d3fc3-6f30-4ff9-8d97-013fb71cdfb7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47672025-04-30T06:46:12.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
47682025-04-30T06:46:12.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90bf2cd1-fd6b-47dc-b539-9c2255a4e73f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47692025-04-30T06:46:12.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
47702025-04-30T06:46:12.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eee1f268-a3f3-4afd-b87c-1a09dd2deda1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47712025-04-30T06:46:12.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
47722025-04-30T06:46:12.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "513fbfd4-e410-499e-8db1-30c1eeaf2420", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47732025-04-30T06:46:12.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
47742025-04-30T06:46:12.307ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47752025-04-30T06:46:12.307ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47762025-04-30T06:46:12.307ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47772025-04-30T06:46:12.307ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47782025-04-30T06:46:12.308ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47792025-04-30T06:46:12.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "591ac2bd-0684-4fa6-ba0a-c150e1281965", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47802025-04-30T06:46:12.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
47812025-04-30T06:46:12.311ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47822025-04-30T06:46:12.319ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47832025-04-30T06:46:12.319ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47842025-04-30T06:46:12.319ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47852025-04-30T06:46:12.320ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47862025-04-30T06:46:12.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb57bcf-d500-4872-aa3c-a45b0e2adffd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47872025-04-30T06:46:12.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
47882025-04-30T06:46:12.323ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47892025-04-30T06:46:12.323ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
47902025-04-30T06:46:12.324ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47912025-04-30T06:46:12.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "798a7e1d-5dd2-4c74-ac1a-ed850aebb40f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47922025-04-30T06:46:12.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
47932025-04-30T06:46:12.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76668834-cc00-4669-8ce7-123a8c0c09bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47942025-04-30T06:46:12.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
47952025-04-30T06:46:12.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b7e583-54aa-4401-831f-e11a060adf2c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47962025-04-30T06:46:12.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
47972025-04-30T06:46:12.330ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c6dffee-ace6-4183-b0df-a57b0ecef69e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
47982025-04-30T06:46:12.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
47992025-04-30T06:46:12.337ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de696669-0278-4eb5-8b94-fabba3a7e17f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48002025-04-30T06:46:12.337ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
48012025-04-30T06:46:12.340ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c45e7b-dbea-4337-aa78-3a3f421db137", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48022025-04-30T06:46:12.340ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
48032025-04-30T06:46:12.344ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e3ae8f-267c-4a01-8d88-044cf44029e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48042025-04-30T06:46:12.344ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
48052025-04-30T06:46:12.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19199ad1-fd43-44a6-a6d0-4ac8951fed3f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48062025-04-30T06:46:12.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
48072025-04-30T06:46:12.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad70643-b100-4ce5-a0db-b45d0ba3468e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48082025-04-30T06:46:12.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
48092025-04-30T06:46:12.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6b3ce94-38b6-4840-a2af-676674dbdc71", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48102025-04-30T06:46:12.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
48112025-04-30T06:46:12.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19e533f-c402-4e9f-8da0-0c26f5ae064f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48122025-04-30T06:46:12.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
48132025-04-30T06:46:12.359ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
48142025-04-30T06:46:12.362ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a73e4401-d959-4a58-8ffe-df8b7dbed272", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48152025-04-30T06:46:12.363ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
48162025-04-30T06:46:12.365ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a065df-c9ed-4cf9-83fd-f0b7a8fd80ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48172025-04-30T06:46:12.365ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
48182025-04-30T06:46:12.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f1b241-c2dc-49dd-9cb5-4fcf7b428335", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48192025-04-30T06:46:12.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
48202025-04-30T06:46:12.369ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88b1ac74-c52e-4466-94ca-63d437783839", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48212025-04-30T06:46:12.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
48222025-04-30T06:46:12.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa2918b4-fba2-42e9-96f9-d99466ac057c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48232025-04-30T06:46:12.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
48242025-04-30T06:46:12.385ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4575486e-70c5-4498-9550-f6a2dd3fec52", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48252025-04-30T06:46:12.385ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
48262025-04-30T06:46:12.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae47e9c-f398-4249-9187-16bc0ee093c4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48272025-04-30T06:46:12.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
48282025-04-30T06:46:12.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "361655c8-5859-416f-91e1-51e5596a4a64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48292025-04-30T06:46:12.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
48302025-04-30T06:46:12.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9370fa5-dcf0-4c6a-b7a1-2e3d40744faa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48312025-04-30T06:46:12.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
48322025-04-30T06:46:12.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bef586a5-3a3a-47d8-80da-7deaf6e03277", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48332025-04-30T06:46:12.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
48342025-04-30T06:46:12.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ecec760-9b81-45eb-b4b7-9b6270a68635", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48352025-04-30T06:46:12.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
48362025-04-30T06:46:12.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e511835a-87ae-4c45-b485-1cad4cf099c1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48372025-04-30T06:46:12.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
48382025-04-30T06:46:12.405ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b36ae3-6472-49b5-babd-972bd81d2df5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48392025-04-30T06:46:12.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
48402025-04-30T06:46:12.408ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e6026c4-4560-452e-91cb-0b9c74dee77a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48412025-04-30T06:46:12.408ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48422025-04-30T06:46:12.409ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48432025-04-30T06:46:12.409ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48442025-04-30T06:46:12.409ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48452025-04-30T06:46:12.409ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48462025-04-30T06:46:12.411ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48472025-04-30T06:46:12.411ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48482025-04-30T06:46:12.411ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48492025-04-30T06:46:12.411ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
48502025-04-30T06:46:12.412ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48512025-04-30T06:46:12.416ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48522025-04-30T06:46:12.430ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48532025-04-30T06:46:12.430ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48542025-04-30T06:46:12.430ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48552025-04-30T06:46:12.430ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48562025-04-30T06:46:12.431ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48572025-04-30T06:46:12.432ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48582025-04-30T06:46:12.438ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48592025-04-30T06:46:12.447ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48602025-04-30T06:46:12.447ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48612025-04-30T06:46:12.447ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
48622025-04-30T06:46:12.448ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48632025-04-30T06:46:12.448ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48642025-04-30T06:46:12.452ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48652025-04-30T06:46:12.452ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48662025-04-30T06:46:12.452ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48672025-04-30T06:46:12.453ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48682025-04-30T06:46:12.471ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48692025-04-30T06:46:12.471ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
48702025-04-30T06:46:12.475ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48712025-04-30T06:46:12.553ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
48722025-04-30T06:46:12.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
48732025-04-30T06:46:12.559ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:63297 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
48742025-04-30T06:46:12.561ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:63297 req_id = 3e0f1e14-beed-48d0-984a-8f3f266b8073 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48752025-04-30T06:46:12.563ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1227 local_addr = [::1]:54590 method = GET remote_addr = [::1]:63297 req_id = 3e0f1e14-beed-48d0-984a-8f3f266b8073 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
48762025-04-30T06:46:12.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e0f1e14-beed-48d0-984a-8f3f266b8073", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:11 GMT"} })
48772025-04-30T06:46:12.570ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48782025-04-30T06:46:12.570ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
48792025-04-30T06:46:12.570ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
48802025-04-30T06:46:12.572ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
48812025-04-30T06:46:12.572ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48822025-04-30T06:46:12.572ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48832025-04-30T06:46:12.572ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48842025-04-30T06:46:12.573ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48852025-04-30T06:46:12.576ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48862025-04-30T06:46:12.580ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
48872025-04-30T06:46:12.580ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48882025-04-30T06:46:12.580ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48892025-04-30T06:46:12.581ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48902025-04-30T06:46:12.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48912025-04-30T06:46:12.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48922025-04-30T06:46:12.641ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48932025-04-30T06:46:12.642ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48942025-04-30T06:46:12.642ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48952025-04-30T06:46:12.642ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
48962025-04-30T06:46:12.644ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
48972025-04-30T06:46:12.656ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
48982025-04-30T06:46:12.656ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
48992025-04-30T06:46:12.656ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 689.275738ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49002025-04-30T06:46:12.656ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49012025-04-30T06:46:12.656ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49022025-04-30T06:46:12.656ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49032025-04-30T06:46:12.657ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49042025-04-30T06:46:12.657ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49052025-04-30T06:46:12.659ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49062025-04-30T06:46:12.662ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49072025-04-30T06:46:12.670ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49082025-04-30T06:46:12.670ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49092025-04-30T06:46:12.670ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49102025-04-30T06:46:12.670ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49112025-04-30T06:46:12.675ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
49122025-04-30T06:46:12.675ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49132025-04-30T06:46:12.676ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 16 2f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ./..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55453 req_id = ff896b73-bc0e-4410-ab72-30f203ee993a
49142025-04-30T06:46:12.676ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5679,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55453 req_id = ff896b73-bc0e-4410-ab72-30f203ee993a
49152025-04-30T06:46:12.676ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
49162025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49172025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55453 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ff896b73-bc0e-4410-ab72-30f203ee993a
49182025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4437, tv_nsec: 476162004 } })
49192025-04-30T06:46:12.677ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
49202025-04-30T06:46:12.677ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6a 05 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e j...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57357 req_id = f4af33dd-2b67-482d-827a-3c4757414b50
49212025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27141,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57357 req_id = f4af33dd-2b67-482d-827a-3c4757414b50
49222025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49232025-04-30T06:46:12.677ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57357 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f4af33dd-2b67-482d-827a-3c4757414b50
49242025-04-30T06:46:12.743ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49252025-04-30T06:46:12.744ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49262025-04-30T06:46:12.744ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49272025-04-30T06:46:12.744ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49282025-04-30T06:46:12.744ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
49292025-04-30T06:46:12.745ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49302025-04-30T06:46:12.746ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3c 62 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c <b..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52496 req_id = fc9865ee-0c6f-462b-9720-a326ed983a29
49312025-04-30T06:46:12.746ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15458,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52496 req_id = fc9865ee-0c6f-462b-9720-a326ed983a29
49322025-04-30T06:46:12.746ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
49332025-04-30T06:46:12.746ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15458, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52496 req_id = fc9865ee-0c6f-462b-9720-a326ed983a29
49342025-04-30T06:46:12.758ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
49352025-04-30T06:46:12.758ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
49362025-04-30T06:46:12.758ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
49372025-04-30T06:46:12.760ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49382025-04-30T06:46:12.768ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49392025-04-30T06:46:12.768ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49402025-04-30T06:46:12.768ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
49412025-04-30T06:46:12.769ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49422025-04-30T06:46:12.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca6304f0-b997-422c-839b-3672dad7843c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49432025-04-30T06:46:12.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
49442025-04-30T06:46:12.774ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49452025-04-30T06:46:12.774ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
49462025-04-30T06:46:12.775ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49472025-04-30T06:46:12.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5271b2d2-57b8-493a-8d34-8df7bb1a4de6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49482025-04-30T06:46:12.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
49492025-04-30T06:46:12.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec647eb7-7492-4d70-aa5a-741ae9069b9b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49502025-04-30T06:46:12.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
49512025-04-30T06:46:12.786ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "214ebcdc-cde9-4a18-b541-0facfeba9e47", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49522025-04-30T06:46:12.786ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
49532025-04-30T06:46:12.789ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f2bdc15-8c17-4ee4-907e-a212c52001c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49542025-04-30T06:46:12.790ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
49552025-04-30T06:46:12.794ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "adaea200-2580-4536-9778-637c6797f0e3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49562025-04-30T06:46:12.794ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
49572025-04-30T06:46:12.797ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "577d844c-710c-490a-a68b-d3fc13fc6b41", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49582025-04-30T06:46:12.797ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
49592025-04-30T06:46:12.800ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e30ceb7e-6214-48d4-9b8f-a263d90bb011", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49602025-04-30T06:46:12.800ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
49612025-04-30T06:46:12.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe709e2-879d-40b5-94f8-8d2d8705c3b2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49622025-04-30T06:46:12.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
49632025-04-30T06:46:12.806ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f92f6c6b-a940-41fe-9927-d178b70d5f94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49642025-04-30T06:46:12.806ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
49652025-04-30T06:46:12.811ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5367d4a1-f7f8-425c-a490-dbc603af5e0f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49662025-04-30T06:46:12.812ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
49672025-04-30T06:46:12.812ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
49682025-04-30T06:46:12.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e776cb2-ec5e-4437-b749-1aa113bfa6d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49692025-04-30T06:46:12.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
49702025-04-30T06:46:12.815ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a12cad6-3c9b-48d3-aaca-3475ff8d991b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49712025-04-30T06:46:12.816ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
49722025-04-30T06:46:12.818ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15086f87-a8ea-4532-9bbb-ce99ea4412a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49732025-04-30T06:46:12.818ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
49742025-04-30T06:46:12.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86c5fdb1-0d7a-4717-a402-b78469a07c54", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49752025-04-30T06:46:12.821ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
49762025-04-30T06:46:12.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab439b8-f5f4-4ab2-adcf-3d85b70c3646", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49772025-04-30T06:46:12.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
49782025-04-30T06:46:12.832ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b4ec98-49b9-461c-9155-74a975d33a65", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49792025-04-30T06:46:12.832ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
49802025-04-30T06:46:12.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ac3f586-01a5-4ee3-9049-f6a2703d03e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49812025-04-30T06:46:12.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
49822025-04-30T06:46:12.838ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cfb4359-64b1-4ea9-b961-67a0d866f940", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49832025-04-30T06:46:12.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
49842025-04-30T06:46:12.840ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e170268-3296-4c8e-9505-bf4077518dd7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49852025-04-30T06:46:12.840ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
49862025-04-30T06:46:12.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0447d16-4c54-4ba6-abc9-4d7a7d223277", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49872025-04-30T06:46:12.859ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
49882025-04-30T06:46:12.863ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49892025-04-30T06:46:12.863ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49902025-04-30T06:46:12.864ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49912025-04-30T06:46:12.864ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
49922025-04-30T06:46:12.865ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49932025-04-30T06:46:12.865ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92cecf90-3ee6-46b8-8d17-c6f90148f6ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
49942025-04-30T06:46:12.866ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
49952025-04-30T06:46:12.869ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49962025-04-30T06:46:12.875ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49972025-04-30T06:46:12.875ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49982025-04-30T06:46:12.875ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
49992025-04-30T06:46:12.876ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50002025-04-30T06:46:12.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4005e5d-648c-4a4f-850b-b3442ed5f1ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50012025-04-30T06:46:12.878ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
50022025-04-30T06:46:12.879ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50032025-04-30T06:46:12.880ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
50042025-04-30T06:46:12.880ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50052025-04-30T06:46:12.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21009295-ba14-4ac4-b2e0-ca71ac7f4f2c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50062025-04-30T06:46:12.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
50072025-04-30T06:46:12.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6afff566-e50d-45e9-93ea-6d8d96ae0f75", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50082025-04-30T06:46:12.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
50092025-04-30T06:46:12.887ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "528c3b95-d2e5-4577-9380-faa5b76dbef6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50102025-04-30T06:46:12.887ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
50112025-04-30T06:46:12.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6719eeea-09ef-4529-b2bf-868733c788d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50122025-04-30T06:46:12.890ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
50132025-04-30T06:46:12.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77ff2f10-798c-4eb5-9c96-b25ff941201a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50142025-04-30T06:46:12.896ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
50152025-04-30T06:46:12.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1595c14-d7ec-417c-b3b6-a9143e74b6b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50162025-04-30T06:46:12.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
50172025-04-30T06:46:12.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a7f8f3-43b1-41f5-abaa-cd09ba083677", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50182025-04-30T06:46:12.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
50192025-04-30T06:46:12.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba86d4e9-f6be-4ced-b5a0-a3aeb4ad60df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50202025-04-30T06:46:12.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
50212025-04-30T06:46:12.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f338efec-42eb-490b-84d0-760dfa53da81", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50222025-04-30T06:46:12.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
50232025-04-30T06:46:12.910ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
50242025-04-30T06:46:12.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec960e9-b02a-4807-b923-ebcc5890892b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50252025-04-30T06:46:12.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
50262025-04-30T06:46:12.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a25e846-33b8-46cc-83b1-241e3ea22191", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50272025-04-30T06:46:12.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
50282025-04-30T06:46:12.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b9ad7e-c8ee-43bd-a6b5-71f3e96ed30f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50292025-04-30T06:46:12.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
50302025-04-30T06:46:12.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71137de9-3821-4faf-ad99-45c1d7a6d697", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50312025-04-30T06:46:12.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
50322025-04-30T06:46:12.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e214fb12-fabd-499f-adb2-668f601ffcae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50332025-04-30T06:46:12.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
50342025-04-30T06:46:12.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "326896c3-de54-4dcc-bcda-62bfbfaabc9b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50352025-04-30T06:46:12.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
50362025-04-30T06:46:12.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f0adc66-f54c-43cd-ba61-f22027295062", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50372025-04-30T06:46:12.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
50382025-04-30T06:46:12.928ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f6baa3b-524e-488d-9ae7-66ce9438a654", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50392025-04-30T06:46:12.928ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
50402025-04-30T06:46:12.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4875f08c-4d17-403c-8f6a-022ad4ca45ee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50412025-04-30T06:46:12.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
50422025-04-30T06:46:12.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b81e2d1-80d9-4661-9560-91a2386a5e44", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50432025-04-30T06:46:12.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
50442025-04-30T06:46:12.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "127e4f22-57f6-41bc-8bb0-7aacaef35e93", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50452025-04-30T06:46:12.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
50462025-04-30T06:46:12.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f38cfd64-f707-4b09-bb38-b5d4ad12cd21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50472025-04-30T06:46:12.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
50482025-04-30T06:46:12.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09307c59-be3a-4109-ba63-d1cbb47ffabc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50492025-04-30T06:46:12.940ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50502025-04-30T06:46:12.940ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50512025-04-30T06:46:12.940ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50522025-04-30T06:46:12.940ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50532025-04-30T06:46:12.940ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50542025-04-30T06:46:12.943ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50552025-04-30T06:46:12.949ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50562025-04-30T06:46:12.949ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50572025-04-30T06:46:12.949ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50582025-04-30T06:46:12.949ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50592025-04-30T06:46:12.949ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50602025-04-30T06:46:12.950ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50612025-04-30T06:46:12.961ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50622025-04-30T06:46:12.961ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50632025-04-30T06:46:12.961ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50642025-04-30T06:46:12.961ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
50652025-04-30T06:46:12.962ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50662025-04-30T06:46:12.963ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50672025-04-30T06:46:12.967ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50682025-04-30T06:46:12.967ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50692025-04-30T06:46:12.967ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50702025-04-30T06:46:12.968ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50712025-04-30T06:46:12.971ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50722025-04-30T06:46:12.995ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50732025-04-30T06:46:12.995ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50742025-04-30T06:46:12.995ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
50752025-04-30T06:46:12.996ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50762025-04-30T06:46:13.083ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
50772025-04-30T06:46:13.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
50782025-04-30T06:46:13.084ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:41586 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
50792025-04-30T06:46:13.085ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50802025-04-30T06:46:13.085ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50812025-04-30T06:46:13.085ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
50822025-04-30T06:46:13.086ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50832025-04-30T06:46:13.086ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50842025-04-30T06:46:13.086ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
50852025-04-30T06:46:13.087ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50862025-04-30T06:46:13.087ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:41586 req_id = ad27d77c-8a5f-42dc-b9d9-51d0b35cc26b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50872025-04-30T06:46:13.088ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 716 local_addr = [::1]:54590 method = GET remote_addr = [::1]:41586 req_id = ad27d77c-8a5f-42dc-b9d9-51d0b35cc26b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
50882025-04-30T06:46:13.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad27d77c-8a5f-42dc-b9d9-51d0b35cc26b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
50892025-04-30T06:46:13.090ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50902025-04-30T06:46:13.090ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50912025-04-30T06:46:13.090ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
50922025-04-30T06:46:13.092ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
50932025-04-30T06:46:13.092ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50942025-04-30T06:46:13.092ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50952025-04-30T06:46:13.092ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50962025-04-30T06:46:13.093ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50972025-04-30T06:46:13.094ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
50982025-04-30T06:46:13.094ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50992025-04-30T06:46:13.097ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51002025-04-30T06:46:13.101ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51012025-04-30T06:46:13.101ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51022025-04-30T06:46:13.101ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51032025-04-30T06:46:13.102ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51042025-04-30T06:46:13.109ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
51052025-04-30T06:46:13.109ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51062025-04-30T06:46:13.147ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
51072025-04-30T06:46:13.153ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51082025-04-30T06:46:13.154ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51092025-04-30T06:46:13.154ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 498.142736ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51102025-04-30T06:46:13.155ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51112025-04-30T06:46:13.155ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51122025-04-30T06:46:13.155ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51132025-04-30T06:46:13.155ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51142025-04-30T06:46:13.155ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51152025-04-30T06:46:13.155ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51162025-04-30T06:46:13.166ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51172025-04-30T06:46:13.171ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51182025-04-30T06:46:13.171ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51192025-04-30T06:46:13.171ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51202025-04-30T06:46:13.172ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51212025-04-30T06:46:13.176ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
51222025-04-30T06:46:13.176ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51232025-04-30T06:46:13.177ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2a e5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 *............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64744 req_id = 3e5154e7-6bd2-47c2-98d2-50c65fc26c20
51242025-04-30T06:46:13.177ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10981,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64744 req_id = 3e5154e7-6bd2-47c2-98d2-50c65fc26c20
51252025-04-30T06:46:13.178ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
51262025-04-30T06:46:13.178ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51272025-04-30T06:46:13.178ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64744 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3e5154e7-6bd2-47c2-98d2-50c65fc26c20
51282025-04-30T06:46:13.179ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4437, tv_nsec: 978057708 } })
51292025-04-30T06:46:13.179ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
51302025-04-30T06:46:13.179ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 95 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53098 req_id = 19e72679-7232-459a-b686-776befe26dbd
51312025-04-30T06:46:13.179ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38239,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53098 req_id = 19e72679-7232-459a-b686-776befe26dbd
51322025-04-30T06:46:13.180ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51332025-04-30T06:46:13.180ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53098 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 19e72679-7232-459a-b686-776befe26dbd
51342025-04-30T06:46:13.233ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51352025-04-30T06:46:13.234ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51362025-04-30T06:46:13.234ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51372025-04-30T06:46:13.234ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51382025-04-30T06:46:13.234ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
51392025-04-30T06:46:13.234ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51402025-04-30T06:46:13.235ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51412025-04-30T06:46:13.235ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e9 c4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62547 req_id = f91cf2a6-9b3d-4a19-b873-36fd1425fb9c
51422025-04-30T06:46:13.236ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59844,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62547 req_id = f91cf2a6-9b3d-4a19-b873-36fd1425fb9c
51432025-04-30T06:46:13.236ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
51442025-04-30T06:46:13.236ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59844, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62547 req_id = f91cf2a6-9b3d-4a19-b873-36fd1425fb9c
51452025-04-30T06:46:13.236ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
51462025-04-30T06:46:13.237ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
51472025-04-30T06:46:13.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
51482025-04-30T06:46:13.238ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51492025-04-30T06:46:13.244ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51502025-04-30T06:46:13.244ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51512025-04-30T06:46:13.244ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
51522025-04-30T06:46:13.245ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51532025-04-30T06:46:13.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "36bada16-5709-4e9a-a253-1e7c6c99f7cd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51542025-04-30T06:46:13.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
51552025-04-30T06:46:13.248ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51562025-04-30T06:46:13.248ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
51572025-04-30T06:46:13.248ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51582025-04-30T06:46:13.250ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e350f863-21f5-42da-8260-82de9cc8e00b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51592025-04-30T06:46:13.250ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
51602025-04-30T06:46:13.255ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cccb193-6f97-4c70-9162-6e11a443644b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51612025-04-30T06:46:13.255ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
51622025-04-30T06:46:13.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b35d75cf-d61f-481b-81a3-7ae3947592b4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51632025-04-30T06:46:13.258ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
51642025-04-30T06:46:13.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f09d2d3-1489-4a31-8817-c6b3b77ed29b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51652025-04-30T06:46:13.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
51662025-04-30T06:46:13.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f312c8-0966-42fe-a329-cc1fefe2d328", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51672025-04-30T06:46:13.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
51682025-04-30T06:46:13.266ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b35d2ba6-b88a-49ca-91b3-15b0190793a0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51692025-04-30T06:46:13.266ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
51702025-04-30T06:46:13.270ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af2b9ce8-08f7-4501-a0fc-1e139169ac60", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51712025-04-30T06:46:13.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
51722025-04-30T06:46:13.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5e6267f-9260-408f-a605-5228d326eec1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51732025-04-30T06:46:13.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
51742025-04-30T06:46:13.275ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9b499ea-797a-4db7-972f-68ec2113d480", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51752025-04-30T06:46:13.275ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
51762025-04-30T06:46:13.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d633694-2ca0-4f77-b023-f372401948e6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51772025-04-30T06:46:13.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
51782025-04-30T06:46:13.279ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
51792025-04-30T06:46:13.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "474fae5b-2b0a-46b3-97f7-163461bc6811", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51802025-04-30T06:46:13.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
51812025-04-30T06:46:13.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bf2c53c-137a-4e7b-8825-03035ac15922", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51822025-04-30T06:46:13.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
51832025-04-30T06:46:13.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de808975-a01d-4e15-82ee-240e63414692", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51842025-04-30T06:46:13.286ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
51852025-04-30T06:46:13.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00869eee-636e-4bc7-90ac-bdfe38ce13ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51862025-04-30T06:46:13.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
51872025-04-30T06:46:13.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0b4e096-08f1-46f2-98c5-10e6b8c49dcd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51882025-04-30T06:46:13.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
51892025-04-30T06:46:13.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a45b8e-3177-479d-a847-a6176c4e5b49", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51902025-04-30T06:46:13.293ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
51912025-04-30T06:46:13.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f3f249b-1438-4759-9db8-714d2d212212", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51922025-04-30T06:46:13.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
51932025-04-30T06:46:13.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "186255f5-2062-4ce5-8515-0f42ea9638c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51942025-04-30T06:46:13.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
51952025-04-30T06:46:13.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "450b64a6-836e-4776-bae6-046a0ea0df56", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51962025-04-30T06:46:13.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
51972025-04-30T06:46:13.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fac11456-13e0-49e5-9f64-39db85525bb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
51982025-04-30T06:46:13.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
51992025-04-30T06:46:13.306ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab38c71-d27f-4ecb-8662-ed6d8243e2a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52002025-04-30T06:46:13.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
52012025-04-30T06:46:13.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d2d6e5c-2fd9-42a1-99fd-d74c6fe8466e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52022025-04-30T06:46:13.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
52032025-04-30T06:46:13.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd1a0b2-5bb8-45da-bdfc-b99b1750d3ed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52042025-04-30T06:46:13.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
52052025-04-30T06:46:13.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "265545e9-24f5-4d85-b746-5c8cfd630290", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52062025-04-30T06:46:13.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
52072025-04-30T06:46:13.319ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1611055d-9889-4551-a7a0-045ad2baed80", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52082025-04-30T06:46:13.319ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
52092025-04-30T06:46:13.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5146f4a8-92cd-40a7-9c3d-d190eea88372", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52102025-04-30T06:46:13.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
52112025-04-30T06:46:13.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd62ee1-7a88-424b-a1fa-22b7d9f74b11", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52122025-04-30T06:46:13.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
52132025-04-30T06:46:13.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42c72946-ec5b-4a35-a503-2cb768d8964e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52142025-04-30T06:46:13.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
52152025-04-30T06:46:13.330ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52162025-04-30T06:46:13.330ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52172025-04-30T06:46:13.330ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52182025-04-30T06:46:13.330ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
52192025-04-30T06:46:13.331ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52202025-04-30T06:46:13.334ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8d43c80-7bb6-41fa-92d3-635e13f0e871", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52212025-04-30T06:46:13.334ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
52222025-04-30T06:46:13.335ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52232025-04-30T06:46:13.344ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52242025-04-30T06:46:13.344ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52252025-04-30T06:46:13.344ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
52262025-04-30T06:46:13.345ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52272025-04-30T06:46:13.348ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52282025-04-30T06:46:13.348ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
52292025-04-30T06:46:13.349ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52302025-04-30T06:46:13.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9ddc53-07d5-40d2-8e62-5b48261a8ffc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52312025-04-30T06:46:13.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
52322025-04-30T06:46:13.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e38f395-62a6-4aa9-9977-cc28b0bce5c9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52332025-04-30T06:46:13.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
52342025-04-30T06:46:13.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2629ea5d-7c7b-45c8-9ab4-3de32d306371", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52352025-04-30T06:46:13.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
52362025-04-30T06:46:13.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e44b84b8-1a1b-4254-9e5a-8a113768778c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52372025-04-30T06:46:13.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
52382025-04-30T06:46:13.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0321865c-a661-4946-b098-062198b5cee0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52392025-04-30T06:46:13.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
52402025-04-30T06:46:13.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4ea8b9e-30af-4ffb-a789-1c62982ac0bb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52412025-04-30T06:46:13.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
52422025-04-30T06:46:13.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ff1f5d-2f7d-477a-8836-d30ada159a40", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52432025-04-30T06:46:13.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
52442025-04-30T06:46:13.364ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc2d238d-3d11-484a-81a2-0ca380da79af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52452025-04-30T06:46:13.364ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
52462025-04-30T06:46:13.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15fa283b-386b-4e5b-821a-8e2eaaeb6bef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52472025-04-30T06:46:13.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
52482025-04-30T06:46:13.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a9beb9e-e344-44ee-b218-4508de754206", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52492025-04-30T06:46:13.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
52502025-04-30T06:46:13.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffdc03f0-9f23-42ec-9e90-4622bfa32190", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52512025-04-30T06:46:13.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
52522025-04-30T06:46:13.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a03824-4582-44a1-9ab4-40255213ce9d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52532025-04-30T06:46:13.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
52542025-04-30T06:46:13.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "100d4bba-e146-4ac5-8e6a-4fdeb4b06687", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52552025-04-30T06:46:13.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
52562025-04-30T06:46:13.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "226b2313-593c-41f5-b953-0261ac2aefc5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52572025-04-30T06:46:13.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
52582025-04-30T06:46:13.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4449e2fa-dd24-40da-8ec5-b3b995de7868", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52592025-04-30T06:46:13.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52602025-04-30T06:46:13.383ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52612025-04-30T06:46:13.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52622025-04-30T06:46:13.383ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52632025-04-30T06:46:13.384ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52642025-04-30T06:46:13.388ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52652025-04-30T06:46:13.392ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52662025-04-30T06:46:13.392ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
52672025-04-30T06:46:13.392ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52682025-04-30T06:46:13.392ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52692025-04-30T06:46:13.393ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52702025-04-30T06:46:13.393ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52712025-04-30T06:46:13.394ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
52722025-04-30T06:46:13.397ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52732025-04-30T06:46:13.402ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
52742025-04-30T06:46:13.403ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52752025-04-30T06:46:13.403ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52762025-04-30T06:46:13.407ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52772025-04-30T06:46:13.483ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
52782025-04-30T06:46:13.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
52792025-04-30T06:46:13.484ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52802025-04-30T06:46:13.484ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52812025-04-30T06:46:13.484ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52822025-04-30T06:46:13.484ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
52832025-04-30T06:46:13.485ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52842025-04-30T06:46:13.486ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:35622 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
52852025-04-30T06:46:13.487ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:35622 req_id = 943cf19b-5fbf-49b2-9c0f-a8c5fa84c69a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52862025-04-30T06:46:13.493ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 5263 local_addr = [::1]:54590 method = GET remote_addr = [::1]:35622 req_id = 943cf19b-5fbf-49b2-9c0f-a8c5fa84c69a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
52872025-04-30T06:46:13.494ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "943cf19b-5fbf-49b2-9c0f-a8c5fa84c69a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
52882025-04-30T06:46:13.497ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
52892025-04-30T06:46:13.497ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
52902025-04-30T06:46:13.497ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
52912025-04-30T06:46:13.503ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
52922025-04-30T06:46:13.503ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52932025-04-30T06:46:13.503ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52942025-04-30T06:46:13.503ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52952025-04-30T06:46:13.514ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52962025-04-30T06:46:13.515ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52972025-04-30T06:46:13.528ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52982025-04-30T06:46:13.528ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52992025-04-30T06:46:13.528ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
53002025-04-30T06:46:13.529ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53012025-04-30T06:46:13.543ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53022025-04-30T06:46:13.558ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53032025-04-30T06:46:13.559ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53042025-04-30T06:46:13.559ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53052025-04-30T06:46:13.559ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53062025-04-30T06:46:13.561ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
53072025-04-30T06:46:13.561ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53082025-04-30T06:46:13.561ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53092025-04-30T06:46:13.561ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
53102025-04-30T06:46:13.562ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53112025-04-30T06:46:13.562ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 17ffacf2-35a9-47a4-a546-5b450a6efe84 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53122025-04-30T06:46:13.566ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 17ffacf2-35a9-47a4-a546-5b450a6efe84 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53132025-04-30T06:46:13.573ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 17ffacf2-35a9-47a4-a546-5b450a6efe84 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53142025-04-30T06:46:13.574ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 17ffacf2-35a9-47a4-a546-5b450a6efe84 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53152025-04-30T06:46:13.586ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 24208 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 17ffacf2-35a9-47a4-a546-5b450a6efe84 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
53162025-04-30T06:46:13.587ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "17ffacf2-35a9-47a4-a546-5b450a6efe84", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
53172025-04-30T06:46:13.587ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53182025-04-30T06:46:13.589ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = bbd93f14-1e07-48d4-93fd-edb6a95a00d6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53192025-04-30T06:46:13.589ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = bbd93f14-1e07-48d4-93fd-edb6a95a00d6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53202025-04-30T06:46:13.589ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = bbd93f14-1e07-48d4-93fd-edb6a95a00d6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53212025-04-30T06:46:13.590ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = bbd93f14-1e07-48d4-93fd-edb6a95a00d6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53222025-04-30T06:46:13.610ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 21323 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = bbd93f14-1e07-48d4-93fd-edb6a95a00d6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
53232025-04-30T06:46:13.611ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd93f14-1e07-48d4-93fd-edb6a95a00d6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:12 GMT"} })
53242025-04-30T06:46:13.611ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
53252025-04-30T06:46:13.611ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53262025-04-30T06:46:13.611ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
53272025-04-30T06:46:13.640ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53282025-04-30T06:46:13.640ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53292025-04-30T06:46:13.641ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53302025-04-30T06:46:13.641ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53312025-04-30T06:46:13.641ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 486.397265ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53322025-04-30T06:46:13.641ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53332025-04-30T06:46:13.641ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53342025-04-30T06:46:13.641ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53352025-04-30T06:46:13.641ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53362025-04-30T06:46:13.641ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53372025-04-30T06:46:13.644ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53382025-04-30T06:46:13.645ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
53392025-04-30T06:46:13.647ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53402025-04-30T06:46:13.648ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53412025-04-30T06:46:13.648ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53422025-04-30T06:46:13.648ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53432025-04-30T06:46:13.648ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53442025-04-30T06:46:13.652ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53452025-04-30T06:46:13.652ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53462025-04-30T06:46:13.652ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53472025-04-30T06:46:13.652ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53482025-04-30T06:46:13.658ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
53492025-04-30T06:46:13.658ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53502025-04-30T06:46:13.659ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 43 3c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 C<..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54403 req_id = 08434dcd-8ded-4c28-8421-54e44701e333
53512025-04-30T06:46:13.659ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17212,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54403 req_id = 08434dcd-8ded-4c28-8421-54e44701e333
53522025-04-30T06:46:13.660ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
53532025-04-30T06:46:13.660ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53542025-04-30T06:46:13.660ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54403 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 08434dcd-8ded-4c28-8421-54e44701e333
53552025-04-30T06:46:13.660ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4438, tv_nsec: 459628126 } })
53562025-04-30T06:46:13.660ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
53572025-04-30T06:46:13.661ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 55 24 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e U$..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56722 req_id = 4aa694a4-abfe-47d1-899f-6dce92c91399
53582025-04-30T06:46:13.661ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21796,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56722 req_id = 4aa694a4-abfe-47d1-899f-6dce92c91399
53592025-04-30T06:46:13.661ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53602025-04-30T06:46:13.661ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56722 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4aa694a4-abfe-47d1-899f-6dce92c91399
53612025-04-30T06:46:13.715ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53622025-04-30T06:46:13.716ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53632025-04-30T06:46:13.716ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53642025-04-30T06:46:13.716ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53652025-04-30T06:46:13.716ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
53662025-04-30T06:46:13.718ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53672025-04-30T06:46:13.718ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d4 80 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62088 req_id = 201c7ed2-5341-4d0f-bf78-2cc89057c7e9
53682025-04-30T06:46:13.718ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54400,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62088 req_id = 201c7ed2-5341-4d0f-bf78-2cc89057c7e9
53692025-04-30T06:46:13.719ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
53702025-04-30T06:46:13.720ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54400, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62088 req_id = 201c7ed2-5341-4d0f-bf78-2cc89057c7e9
53712025-04-30T06:46:13.720ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
53722025-04-30T06:46:13.720ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
53732025-04-30T06:46:13.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
53742025-04-30T06:46:13.721ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53752025-04-30T06:46:13.731ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53762025-04-30T06:46:13.731ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53772025-04-30T06:46:13.731ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
53782025-04-30T06:46:13.732ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53792025-04-30T06:46:13.735ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a772cdec-bc86-4c73-b995-a40aed76d67a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53802025-04-30T06:46:13.736ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
53812025-04-30T06:46:13.737ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53822025-04-30T06:46:13.737ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
53832025-04-30T06:46:13.738ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53842025-04-30T06:46:13.743ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83bfcb30-8cbb-49f5-b11a-8ca5c4aecd3a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53852025-04-30T06:46:13.744ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
53862025-04-30T06:46:13.749ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b7d3bc5-8497-44ca-998a-941c87aa0aa5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53872025-04-30T06:46:13.749ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
53882025-04-30T06:46:13.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c0f7748-4d99-4d17-b583-4433674d55bd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53892025-04-30T06:46:13.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
53902025-04-30T06:46:13.759ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25b0a5be-6f8b-4b21-b01e-705345c1ed1f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53912025-04-30T06:46:13.759ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
53922025-04-30T06:46:13.764ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b91d4c32-7de6-4885-bf46-89fbd8593527", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53932025-04-30T06:46:13.764ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
53942025-04-30T06:46:13.766ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09a067b9-7dc3-45d1-9d4e-db94e7f99a84", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53952025-04-30T06:46:13.766ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
53962025-04-30T06:46:13.768ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbc89502-5033-4ce0-b3af-f6b0ed2c4fa2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53972025-04-30T06:46:13.768ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
53982025-04-30T06:46:13.769ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fc3c6e0-96f6-4587-aa20-bb98d9a2269b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
53992025-04-30T06:46:13.769ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
54002025-04-30T06:46:13.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ce475cb-af76-42a4-9c9a-455ff644e9cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54012025-04-30T06:46:13.772ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
54022025-04-30T06:46:13.773ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4db89efb-fd53-449c-a0b2-d5545f8e5422", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54032025-04-30T06:46:13.773ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
54042025-04-30T06:46:13.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9a73e65-3293-4e74-9883-0e1f2dd5b801", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54052025-04-30T06:46:13.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
54062025-04-30T06:46:13.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1adbad6a-49cc-4165-8b85-6471a4be8cb9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54072025-04-30T06:46:13.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
54082025-04-30T06:46:13.779ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85af77eb-d9ea-4e89-8f40-431b1979203b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54092025-04-30T06:46:13.779ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
54102025-04-30T06:46:13.781ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "394f38e8-82f1-4de5-b6cc-9c1b441256c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54112025-04-30T06:46:13.781ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
54122025-04-30T06:46:13.793ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
54132025-04-30T06:46:13.796ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "491d198d-2537-4b78-b36c-e534a1f3674e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54142025-04-30T06:46:13.815ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
54152025-04-30T06:46:13.817ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd819800-a4fc-4ca3-a6b3-7dbfacef7193", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54162025-04-30T06:46:13.817ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
54172025-04-30T06:46:13.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3118486b-d4b6-4a7c-a6d5-30f7fcaf9dcc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54182025-04-30T06:46:13.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
54192025-04-30T06:46:13.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "067a5141-386d-404f-a98d-16b5eec33223", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54202025-04-30T06:46:13.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
54212025-04-30T06:46:13.826ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dbe527a-4fd3-41b5-929b-9e648ea8c6aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54222025-04-30T06:46:13.826ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
54232025-04-30T06:46:13.827ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bfd0a1f-51f2-4b50-a86b-1d5ad11894d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54242025-04-30T06:46:13.828ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
54252025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6439c2d-5421-4793-8e26-55333bf90653", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54262025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
54272025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eab1742c-d04e-424f-a89b-39ed8fc2ecb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54282025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
54292025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac7c1106-fa16-4f3c-91ac-90db26d2bf6b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54302025-04-30T06:46:13.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
54312025-04-30T06:46:13.864ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54322025-04-30T06:46:13.864ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54332025-04-30T06:46:13.865ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54342025-04-30T06:46:13.865ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
54352025-04-30T06:46:13.866ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54362025-04-30T06:46:13.868ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-04-30T06:46:13.874ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54382025-04-30T06:46:13.874ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54392025-04-30T06:46:13.874ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
54402025-04-30T06:46:13.874ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54412025-04-30T06:46:13.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc24526-9d6a-4f69-8b59-038b0fa4e776", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54422025-04-30T06:46:13.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
54432025-04-30T06:46:13.878ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54442025-04-30T06:46:13.878ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
54452025-04-30T06:46:13.878ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54462025-04-30T06:46:13.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20408f7d-e84c-414d-ba57-6bf5e625ad8a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54472025-04-30T06:46:13.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
54482025-04-30T06:46:13.886ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "678b36e6-4bbd-4fe5-b425-ca745c30a913", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54492025-04-30T06:46:13.886ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
54502025-04-30T06:46:13.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e8ca51-acae-4c86-998b-3b40bfc512cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54512025-04-30T06:46:13.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
54522025-04-30T06:46:13.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b7997bd-e0f4-4960-8d9f-c027b7d21c8a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54532025-04-30T06:46:13.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
54542025-04-30T06:46:13.893ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b99395-2997-44f1-9c2a-57497b6f2e20", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54552025-04-30T06:46:13.893ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
54562025-04-30T06:46:13.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "236eba58-a616-4ba8-b735-e88b982684dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54572025-04-30T06:46:13.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
54582025-04-30T06:46:13.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "557a0945-c50a-4790-a36a-b70fca1c74a0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54592025-04-30T06:46:13.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
54602025-04-30T06:46:13.898ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1507854a-dd54-4b47-8e2f-2c9afffc9afa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54612025-04-30T06:46:13.898ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
54622025-04-30T06:46:13.901ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d58b3eef-4daf-4f86-a711-dc022b98bd31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54632025-04-30T06:46:13.901ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
54642025-04-30T06:46:13.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "158fb754-d16e-4eb2-99e2-25f13e4f8b67", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54652025-04-30T06:46:13.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
54662025-04-30T06:46:13.904ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a09abd41-429a-4493-b258-532cd18e5ca1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54672025-04-30T06:46:13.904ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
54682025-04-30T06:46:13.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1584e71-f573-4c64-8e26-ca7a01036075", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54692025-04-30T06:46:13.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
54702025-04-30T06:46:13.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f8d69e-fefc-40be-bc14-e24372754bb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54712025-04-30T06:46:13.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
54722025-04-30T06:46:13.913ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
54732025-04-30T06:46:13.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "514f2314-8513-4f3a-97af-6436d9c2ee1f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54742025-04-30T06:46:13.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
54752025-04-30T06:46:13.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29a2f825-5303-42c9-801e-539acd0cc37d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54762025-04-30T06:46:13.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
54772025-04-30T06:46:13.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0633c5d-e83f-45df-b004-9660ba2777cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54782025-04-30T06:46:13.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
54792025-04-30T06:46:13.922ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94815d8a-95f9-4c65-8568-1b51516564e3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54802025-04-30T06:46:13.922ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
54812025-04-30T06:46:13.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fc5e926-d39d-48fb-97fd-73adcaaf2eb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54822025-04-30T06:46:13.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
54832025-04-30T06:46:13.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b72afee-af67-4fae-a435-1c59808b9494", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54842025-04-30T06:46:13.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
54852025-04-30T06:46:13.925ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e004753a-4d6f-41b4-bd09-cbd6214e5cd8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
54862025-04-30T06:46:13.925ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
54872025-04-30T06:46:13.925ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54882025-04-30T06:46:13.925ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54892025-04-30T06:46:13.925ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54902025-04-30T06:46:13.926ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54912025-04-30T06:46:13.929ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54922025-04-30T06:46:13.936ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
54932025-04-30T06:46:13.936ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
54942025-04-30T06:46:13.936ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54952025-04-30T06:46:13.936ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54962025-04-30T06:46:13.936ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54972025-04-30T06:46:13.938ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54982025-04-30T06:46:13.977ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54992025-04-30T06:46:13.977ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55002025-04-30T06:46:13.977ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55012025-04-30T06:46:13.977ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55022025-04-30T06:46:13.979ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55032025-04-30T06:46:13.979ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55042025-04-30T06:46:13.984ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55052025-04-30T06:46:13.985ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55062025-04-30T06:46:13.985ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55072025-04-30T06:46:13.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55082025-04-30T06:46:13.986ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55092025-04-30T06:46:13.986ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55102025-04-30T06:46:13.987ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
55112025-04-30T06:46:13.988ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
55122025-04-30T06:46:13.989ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55132025-04-30T06:46:14.033ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55142025-04-30T06:46:14.033ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55152025-04-30T06:46:14.033ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55162025-04-30T06:46:14.034ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55172025-04-30T06:46:14.091ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
55182025-04-30T06:46:14.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
55192025-04-30T06:46:14.092ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:62679 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55202025-04-30T06:46:14.095ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
55212025-04-30T06:46:14.103ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55222025-04-30T06:46:14.103ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55232025-04-30T06:46:14.106ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55242025-04-30T06:46:14.106ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
55252025-04-30T06:46:14.106ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
55262025-04-30T06:46:14.107ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:62679 req_id = 1234c913-a814-43b9-9cf2-d7f048e5f6e6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55272025-04-30T06:46:14.109ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2027 local_addr = [::1]:54590 method = GET remote_addr = [::1]:62679 req_id = 1234c913-a814-43b9-9cf2-d7f048e5f6e6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55282025-04-30T06:46:14.110ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1234c913-a814-43b9-9cf2-d7f048e5f6e6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
55292025-04-30T06:46:14.114ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55302025-04-30T06:46:14.115ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55312025-04-30T06:46:14.115ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
55322025-04-30T06:46:14.118ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55332025-04-30T06:46:14.119ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55342025-04-30T06:46:14.119ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55352025-04-30T06:46:14.119ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55362025-04-30T06:46:14.122ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55372025-04-30T06:46:14.132ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
55382025-04-30T06:46:14.132ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55392025-04-30T06:46:14.134ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55402025-04-30T06:46:14.147ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55412025-04-30T06:46:14.148ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55422025-04-30T06:46:14.148ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55432025-04-30T06:46:14.148ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55442025-04-30T06:46:14.211ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
55452025-04-30T06:46:14.213ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55462025-04-30T06:46:14.213ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55472025-04-30T06:46:14.213ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 571.956385ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55482025-04-30T06:46:14.213ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
55492025-04-30T06:46:14.213ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55502025-04-30T06:46:14.213ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55512025-04-30T06:46:14.213ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55522025-04-30T06:46:14.213ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55532025-04-30T06:46:14.214ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55542025-04-30T06:46:14.217ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55552025-04-30T06:46:14.222ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55562025-04-30T06:46:14.222ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55572025-04-30T06:46:14.222ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55582025-04-30T06:46:14.223ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55592025-04-30T06:46:14.229ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
55602025-04-30T06:46:14.229ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55612025-04-30T06:46:14.230ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1e 65 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .e..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56153 req_id = 7b5ca96c-adeb-4ff1-a03e-0c2ed7bff74a
55622025-04-30T06:46:14.236ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7781,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56153 req_id = 7b5ca96c-adeb-4ff1-a03e-0c2ed7bff74a
55632025-04-30T06:46:14.237ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
55642025-04-30T06:46:14.237ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55652025-04-30T06:46:14.238ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56153 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7b5ca96c-adeb-4ff1-a03e-0c2ed7bff74a
55662025-04-30T06:46:14.238ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4439, tv_nsec: 37511721 } })
55672025-04-30T06:46:14.238ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
55682025-04-30T06:46:14.239ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 90 ee 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60286 req_id = 539394ca-6838-4aac-bb39-1e43d4217e88
55692025-04-30T06:46:14.239ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37102,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60286 req_id = 539394ca-6838-4aac-bb39-1e43d4217e88
55702025-04-30T06:46:14.239ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55712025-04-30T06:46:14.239ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60286 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 539394ca-6838-4aac-bb39-1e43d4217e88
55722025-04-30T06:46:14.301ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55732025-04-30T06:46:14.329ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55742025-04-30T06:46:14.329ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55752025-04-30T06:46:14.330ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55762025-04-30T06:46:14.332ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55772025-04-30T06:46:14.333ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55782025-04-30T06:46:14.334ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f 7a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .z..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49169 req_id = ae6431c5-689d-41c2-a292-b24fb3a21e82
55792025-04-30T06:46:14.334ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40826,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49169 req_id = ae6431c5-689d-41c2-a292-b24fb3a21e82
55802025-04-30T06:46:14.334ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
55812025-04-30T06:46:14.335ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40826, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49169 req_id = ae6431c5-689d-41c2-a292-b24fb3a21e82
55822025-04-30T06:46:14.335ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
55832025-04-30T06:46:14.336ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
55842025-04-30T06:46:14.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
55852025-04-30T06:46:14.337ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55862025-04-30T06:46:14.347ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55872025-04-30T06:46:14.348ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55882025-04-30T06:46:14.348ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55892025-04-30T06:46:14.349ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55902025-04-30T06:46:14.356ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "49276d17-d982-4051-9749-1e9214ef10b6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
55912025-04-30T06:46:14.356ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
55922025-04-30T06:46:14.358ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55932025-04-30T06:46:14.358ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
55942025-04-30T06:46:14.359ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55952025-04-30T06:46:14.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b48d80e-2da0-44d4-9fe1-1fc9f874745b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
55962025-04-30T06:46:14.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
55972025-04-30T06:46:14.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "559bd085-686f-4faf-9dbe-40ad44357662", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
55982025-04-30T06:46:14.371ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
55992025-04-30T06:46:14.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1e2ee45-0566-4715-a093-b8e973cb5f41", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56002025-04-30T06:46:14.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
56012025-04-30T06:46:14.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d024587-e238-4262-85ef-09a456607fd0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56022025-04-30T06:46:14.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
56032025-04-30T06:46:14.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74916928-2b87-4e92-8d7f-1a66c80ad16d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56042025-04-30T06:46:14.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
56052025-04-30T06:46:14.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd20f19d-32df-40fe-a199-257600b0d681", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56062025-04-30T06:46:14.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
56072025-04-30T06:46:14.384ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c531959d-bf60-47a0-80bb-111f104f90fa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56082025-04-30T06:46:14.384ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
56092025-04-30T06:46:14.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb1fbb2-0ee5-4541-8d5b-96558813bab0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56102025-04-30T06:46:14.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
56112025-04-30T06:46:14.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad229269-f96c-414e-8b1d-c58d932ce475", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56122025-04-30T06:46:14.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
56132025-04-30T06:46:14.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66753b75-29f1-4b41-8043-1dd5f81f291e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56142025-04-30T06:46:14.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
56152025-04-30T06:46:14.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6abd5062-907b-4624-8e36-27558bdb9d1b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56162025-04-30T06:46:14.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
56172025-04-30T06:46:14.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f465625-e062-4c19-b4fd-fce62be35c43", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56182025-04-30T06:46:14.398ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
56192025-04-30T06:46:14.399ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
56202025-04-30T06:46:14.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a42f824a-22d7-48bb-8371-40ee6cc43ae5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56212025-04-30T06:46:14.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
56222025-04-30T06:46:14.403ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e7bf96a-869e-4a48-bb01-81e011282596", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56232025-04-30T06:46:14.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
56242025-04-30T06:46:14.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f70fda-e814-46b9-97cc-af26429d22f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56252025-04-30T06:46:14.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
56262025-04-30T06:46:14.450ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56272025-04-30T06:46:14.450ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56282025-04-30T06:46:14.450ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56292025-04-30T06:46:14.450ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
56302025-04-30T06:46:14.464ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56312025-04-30T06:46:14.475ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ba874b7-f364-4f7a-9f42-63f963986720", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56322025-04-30T06:46:14.476ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
56332025-04-30T06:46:14.479ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56342025-04-30T06:46:14.486ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56352025-04-30T06:46:14.486ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56362025-04-30T06:46:14.486ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
56372025-04-30T06:46:14.487ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56382025-04-30T06:46:14.491ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b0e8db-f4d0-4049-955f-97849897a2c8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56392025-04-30T06:46:14.491ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
56402025-04-30T06:46:14.491ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56412025-04-30T06:46:14.491ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
56422025-04-30T06:46:14.492ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56432025-04-30T06:46:14.495ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f83d7a08-5ad5-447e-a9c1-77eb9d6d82d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56442025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
56452025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16789123-91fd-48eb-b931-e39036399ab6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56462025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
56472025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db208f8c-a22b-40ae-97f3-783b3f11bd6a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56482025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
56492025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c699e5ee-6575-49ea-99bc-47284d38567d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56502025-04-30T06:46:14.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
56512025-04-30T06:46:14.509ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aff8f27-b738-41e8-8228-74788861fc18", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56522025-04-30T06:46:14.509ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
56532025-04-30T06:46:14.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20147506-3e39-4f55-bda9-5542874b7857", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56542025-04-30T06:46:14.511ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
56552025-04-30T06:46:14.527ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f807260-f169-43b4-ae8d-548c8f3453a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56562025-04-30T06:46:14.528ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
56572025-04-30T06:46:14.530ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70da63bb-ba24-44fc-84dc-deb32dc02f23", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56582025-04-30T06:46:14.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
56592025-04-30T06:46:14.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "341505cc-3976-4632-a8fc-7a83faf89a45", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56602025-04-30T06:46:14.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
56612025-04-30T06:46:14.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4d3e3b7-bb91-459a-bda8-28495924a05b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56622025-04-30T06:46:14.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
56632025-04-30T06:46:14.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "688f1727-69fd-4484-ab9e-019bb95ae5e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56642025-04-30T06:46:14.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
56652025-04-30T06:46:14.541ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
56662025-04-30T06:46:14.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe42e88-8643-4b8b-82a7-7afb221f8cea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56672025-04-30T06:46:14.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
56682025-04-30T06:46:14.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea199998-b6ec-4323-9bf5-9b7f257e0f4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56692025-04-30T06:46:14.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
56702025-04-30T06:46:14.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf879228-0f34-4ff6-8207-eca5e733c834", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56712025-04-30T06:46:14.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
56722025-04-30T06:46:14.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b76f2f4c-50a0-43b4-9a1d-c57dbda0b568", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56732025-04-30T06:46:14.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
56742025-04-30T06:46:14.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38b303b3-d84f-4ab2-b37b-e41b851a8ae3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56752025-04-30T06:46:14.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
56762025-04-30T06:46:14.548ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "337bffb5-0638-4f53-8428-e0436131bc73", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56772025-04-30T06:46:14.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
56782025-04-30T06:46:14.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c296b62e-265b-48b6-ac9e-6499786444fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56792025-04-30T06:46:14.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
56802025-04-30T06:46:14.553ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b3b41c7-64e8-4551-88ef-5a12ba72e5d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56812025-04-30T06:46:14.553ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
56822025-04-30T06:46:14.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa157614-4f8e-46b9-874e-587ad7eeada8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56832025-04-30T06:46:14.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
56842025-04-30T06:46:14.556ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d96941da-5590-411b-b1a2-4a79fb8b7b3c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56852025-04-30T06:46:14.556ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
56862025-04-30T06:46:14.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf463ed-ddb6-406d-b596-1db83764453d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56872025-04-30T06:46:14.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
56882025-04-30T06:46:14.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b272aa7-d43c-442b-a258-8f4eecd444fa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56892025-04-30T06:46:14.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
56902025-04-30T06:46:14.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbcb8a09-c3ce-4fb1-927b-e36e78a8cf3f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56912025-04-30T06:46:14.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
56922025-04-30T06:46:14.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d46a4fa-6a53-40cf-ac67-f286d8acb1df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56932025-04-30T06:46:14.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
56942025-04-30T06:46:14.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4268b2e6-00f9-4353-a0aa-85737dd14645", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56952025-04-30T06:46:14.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
56962025-04-30T06:46:14.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de312dcd-7942-4c9a-9014-6ac251c4221d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:13 GMT"} })
56972025-04-30T06:46:14.574ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56982025-04-30T06:46:14.574ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56992025-04-30T06:46:14.574ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57002025-04-30T06:46:14.574ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57012025-04-30T06:46:14.574ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57022025-04-30T06:46:14.577ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57032025-04-30T06:46:14.586ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57042025-04-30T06:46:14.587ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57052025-04-30T06:46:14.587ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57062025-04-30T06:46:14.587ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57072025-04-30T06:46:14.587ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57082025-04-30T06:46:14.588ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57092025-04-30T06:46:14.592ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57102025-04-30T06:46:14.596ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57112025-04-30T06:46:14.596ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57122025-04-30T06:46:14.596ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57132025-04-30T06:46:14.597ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57142025-04-30T06:46:14.599ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57152025-04-30T06:46:14.599ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57162025-04-30T06:46:14.599ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57172025-04-30T06:46:14.600ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
57182025-04-30T06:46:14.600ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57192025-04-30T06:46:14.603ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57202025-04-30T06:46:14.609ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57212025-04-30T06:46:14.609ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57222025-04-30T06:46:14.609ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
57232025-04-30T06:46:14.609ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57242025-04-30T06:46:14.648ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
57252025-04-30T06:46:14.649ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
57262025-04-30T06:46:14.651ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:51712 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57272025-04-30T06:46:14.651ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57282025-04-30T06:46:14.651ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57292025-04-30T06:46:14.652ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:51712 req_id = e17983a1-cd13-4e12-b5e8-1943a777fa3d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57302025-04-30T06:46:14.655ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2905 local_addr = [::1]:54590 method = GET remote_addr = [::1]:51712 req_id = e17983a1-cd13-4e12-b5e8-1943a777fa3d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57312025-04-30T06:46:14.655ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57322025-04-30T06:46:14.655ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
57332025-04-30T06:46:14.656ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57342025-04-30T06:46:14.657ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57352025-04-30T06:46:14.658ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57362025-04-30T06:46:14.658ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57372025-04-30T06:46:14.658ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57382025-04-30T06:46:14.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e17983a1-cd13-4e12-b5e8-1943a777fa3d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
57392025-04-30T06:46:14.660ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57402025-04-30T06:46:14.660ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57412025-04-30T06:46:14.660ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
57422025-04-30T06:46:14.663ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57432025-04-30T06:46:14.663ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57442025-04-30T06:46:14.663ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57452025-04-30T06:46:14.663ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57462025-04-30T06:46:14.664ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57472025-04-30T06:46:14.670ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57482025-04-30T06:46:14.680ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57492025-04-30T06:46:14.682ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57502025-04-30T06:46:14.682ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57512025-04-30T06:46:14.683ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57522025-04-30T06:46:14.724ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57532025-04-30T06:46:14.725ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57542025-04-30T06:46:14.725ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 512.44807ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57552025-04-30T06:46:14.725ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
57562025-04-30T06:46:14.725ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57572025-04-30T06:46:14.726ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57582025-04-30T06:46:14.726ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57592025-04-30T06:46:14.726ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57602025-04-30T06:46:14.727ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57612025-04-30T06:46:14.733ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57622025-04-30T06:46:14.737ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57632025-04-30T06:46:14.737ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57642025-04-30T06:46:14.737ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57652025-04-30T06:46:14.737ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57662025-04-30T06:46:14.738ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
57672025-04-30T06:46:14.743ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
57682025-04-30T06:46:14.743ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
57692025-04-30T06:46:14.744ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c1 42 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .B..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53160 req_id = e2c2e9f3-c4d4-418b-9c2d-fd19713bc292
57702025-04-30T06:46:14.744ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49474,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53160 req_id = e2c2e9f3-c4d4-418b-9c2d-fd19713bc292
57712025-04-30T06:46:14.744ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
57722025-04-30T06:46:14.744ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57732025-04-30T06:46:14.745ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53160 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e2c2e9f3-c4d4-418b-9c2d-fd19713bc292
57742025-04-30T06:46:14.745ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4439, tv_nsec: 544378690 } })
57752025-04-30T06:46:14.745ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
57762025-04-30T06:46:14.745ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a0 4d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .M..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53943 req_id = 38d935e4-45b9-462a-b514-0efb6fe01ffa
57772025-04-30T06:46:14.745ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53943 req_id = 38d935e4-45b9-462a-b514-0efb6fe01ffa
57782025-04-30T06:46:14.746ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57792025-04-30T06:46:14.746ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53943 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 38d935e4-45b9-462a-b514-0efb6fe01ffa
57802025-04-30T06:46:14.792ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
57812025-04-30T06:46:14.792ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57822025-04-30T06:46:14.793ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57832025-04-30T06:46:14.793ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57842025-04-30T06:46:14.793ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
57852025-04-30T06:46:14.793ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57862025-04-30T06:46:14.794ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 24 c4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c $............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63984 req_id = 699f04e5-da0f-4570-b107-e415112e55ca
57872025-04-30T06:46:14.795ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9412,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63984 req_id = 699f04e5-da0f-4570-b107-e415112e55ca
57882025-04-30T06:46:14.795ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
57892025-04-30T06:46:14.795ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9412, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63984 req_id = 699f04e5-da0f-4570-b107-e415112e55ca
57902025-04-30T06:46:14.795ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
57912025-04-30T06:46:14.795ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
57922025-04-30T06:46:14.796ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
57932025-04-30T06:46:14.803ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57942025-04-30T06:46:14.825ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57952025-04-30T06:46:14.825ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57962025-04-30T06:46:14.825ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
57972025-04-30T06:46:14.825ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57982025-04-30T06:46:14.838ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d50c8cf8-5895-4d02-9f50-5fa21d4dcffc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
57992025-04-30T06:46:14.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
58002025-04-30T06:46:14.840ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58012025-04-30T06:46:14.840ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
58022025-04-30T06:46:14.840ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58032025-04-30T06:46:14.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfc23f44-d51e-4de4-b9f2-a485ac82e19a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58042025-04-30T06:46:14.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
58052025-04-30T06:46:14.859ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50798074-f895-4ead-94c8-a6007b6d5332", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58062025-04-30T06:46:14.859ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
58072025-04-30T06:46:14.866ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc110a17-7909-4a79-baef-a227a5e4814c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58082025-04-30T06:46:14.866ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
58092025-04-30T06:46:14.871ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3648c323-a5e6-47f8-850d-851e3420f02a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58102025-04-30T06:46:14.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
58112025-04-30T06:46:14.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c80c2b-ea09-4fbb-8918-a5f982bf46fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58122025-04-30T06:46:14.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
58132025-04-30T06:46:14.898ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
58142025-04-30T06:46:14.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec35ad6-5430-4c84-9299-b2256f655e90", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58152025-04-30T06:46:14.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
58162025-04-30T06:46:14.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b18ba4fc-5930-472b-9924-bf0a05a38397", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58172025-04-30T06:46:14.904ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
58182025-04-30T06:46:14.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "740171d0-22d7-45e6-8028-b4044973202a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58192025-04-30T06:46:14.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
58202025-04-30T06:46:14.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee464e3-fa79-40e5-abc4-951f3f7e43f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58212025-04-30T06:46:14.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
58222025-04-30T06:46:14.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "220049dd-b989-403a-9bb0-b2505282228b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58232025-04-30T06:46:14.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
58242025-04-30T06:46:14.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12525da1-9770-452d-b14c-9b814729e8ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58252025-04-30T06:46:14.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
58262025-04-30T06:46:14.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "852be693-8d18-4186-bc93-bb4422936275", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58272025-04-30T06:46:14.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
58282025-04-30T06:46:14.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7a4fa1a-1382-47e3-aa9a-20a60c295e3d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58292025-04-30T06:46:14.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
58302025-04-30T06:46:14.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06c7f8a5-73b1-436d-bc5d-dfd36191d506", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58312025-04-30T06:46:14.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
58322025-04-30T06:46:14.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70062948-3b6f-4810-befc-4b6ceaa635b4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58332025-04-30T06:46:14.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
58342025-04-30T06:46:14.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e865c37d-0d51-41fd-8d60-67287c089cbe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58352025-04-30T06:46:14.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
58362025-04-30T06:46:14.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd0b8cbf-54a9-43a6-91a1-75dda724608d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58372025-04-30T06:46:14.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
58382025-04-30T06:46:14.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "591d56fe-f329-44c1-8160-d8b0fccd1aa7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58392025-04-30T06:46:14.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
58402025-04-30T06:46:14.946ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7751822e-a64d-4770-a353-5fd0cbe6462f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58412025-04-30T06:46:14.947ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
58422025-04-30T06:46:14.948ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17bcaa15-24dd-4c2c-9658-e5154efcae8c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58432025-04-30T06:46:14.948ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
58442025-04-30T06:46:14.949ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58452025-04-30T06:46:14.949ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58462025-04-30T06:46:14.950ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58472025-04-30T06:46:14.950ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
58482025-04-30T06:46:14.950ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58492025-04-30T06:46:14.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15077f6d-ce56-4027-9dc4-6fbcef3fbb99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58502025-04-30T06:46:14.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
58512025-04-30T06:46:14.954ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58522025-04-30T06:46:14.959ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58532025-04-30T06:46:14.977ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58542025-04-30T06:46:14.977ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
58552025-04-30T06:46:14.978ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58562025-04-30T06:46:14.979ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6eda40f-65dd-45ce-ba46-6cc99f8ce279", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58572025-04-30T06:46:14.979ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
58582025-04-30T06:46:14.980ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58592025-04-30T06:46:14.980ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58602025-04-30T06:46:14.980ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
58612025-04-30T06:46:14.981ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58622025-04-30T06:46:14.983ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c29643ae-ead9-4be7-addc-86aeae5b884f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58632025-04-30T06:46:14.984ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
58642025-04-30T06:46:14.984ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58652025-04-30T06:46:14.984ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
58662025-04-30T06:46:15.005ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58672025-04-30T06:46:15.016ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
58682025-04-30T06:46:15.016ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58692025-04-30T06:46:15.018ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0b67e2e-8a2f-4d94-87e2-d99d8bc01014", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58702025-04-30T06:46:15.018ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
58712025-04-30T06:46:15.069ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
58722025-04-30T06:46:15.069ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58732025-04-30T06:46:15.083ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58742025-04-30T06:46:15.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bdca6e2-4c6c-4957-a5c5-4ac1d33d5434", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58752025-04-30T06:46:15.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
58762025-04-30T06:46:15.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "310f0d33-b4ea-45e9-a74d-50dcd8ff367c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58772025-04-30T06:46:15.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
58782025-04-30T06:46:15.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fca790e-adf0-4898-8bd0-05e0de7c4310", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58792025-04-30T06:46:15.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
58802025-04-30T06:46:15.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87017311-c521-4901-8359-d5a8136f8a30", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58812025-04-30T06:46:15.102ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
58822025-04-30T06:46:15.108ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
58832025-04-30T06:46:15.116ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4125e475-4982-4266-b076-7f45d4b65dda", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58842025-04-30T06:46:15.117ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
58852025-04-30T06:46:15.118ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "693b5483-256b-44bd-b35e-d4aaec1e7ce6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58862025-04-30T06:46:15.118ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
58872025-04-30T06:46:15.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9503a688-c287-4051-850f-477f4a9bb5cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58882025-04-30T06:46:15.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
58892025-04-30T06:46:15.151ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f484a959-3c95-477b-94a1-49b074fd54d3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58902025-04-30T06:46:15.151ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
58912025-04-30T06:46:15.153ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85dca1a-042a-4922-b50f-d58d18b72e86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58922025-04-30T06:46:15.153ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
58932025-04-30T06:46:15.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "319bfe4a-0ba9-4565-a18f-ae97248e9a77", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58942025-04-30T06:46:15.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
58952025-04-30T06:46:15.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c42370fd-a808-4ba4-86fd-10c860704441", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58962025-04-30T06:46:15.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
58972025-04-30T06:46:15.159ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67667e6-18da-4887-a197-24957c0a0e28", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
58982025-04-30T06:46:15.159ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
58992025-04-30T06:46:15.159ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59002025-04-30T06:46:15.159ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59012025-04-30T06:46:15.159ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59022025-04-30T06:46:15.159ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59032025-04-30T06:46:15.160ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59042025-04-30T06:46:15.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdb4957d-ce81-4fb9-acfa-ff2c1bf9ca99", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59052025-04-30T06:46:15.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
59062025-04-30T06:46:15.163ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59072025-04-30T06:46:15.169ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59082025-04-30T06:46:15.169ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59092025-04-30T06:46:15.169ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59102025-04-30T06:46:15.169ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59112025-04-30T06:46:15.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bef3053e-2d6a-42b9-b903-853b3b7e6aa2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59122025-04-30T06:46:15.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
59132025-04-30T06:46:15.172ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59142025-04-30T06:46:15.174ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59152025-04-30T06:46:15.174ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59162025-04-30T06:46:15.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc3fdd1-1ac7-4aae-bcdf-e2df8c34ad6c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59172025-04-30T06:46:15.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
59182025-04-30T06:46:15.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da01eb39-73d1-4806-a91d-9c65355e8f1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59192025-04-30T06:46:15.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
59202025-04-30T06:46:15.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f4f3393-1b95-4929-ba2b-678772e61e82", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59212025-04-30T06:46:15.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
59222025-04-30T06:46:15.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "395fd449-4b5e-43ac-a50b-72ce123dc2e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59232025-04-30T06:46:15.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
59242025-04-30T06:46:15.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde09bf8-1992-4305-b755-d0e60565cc4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59252025-04-30T06:46:15.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
59262025-04-30T06:46:15.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe9ee67a-057d-408c-875e-5b677666cfed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59272025-04-30T06:46:15.195ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59282025-04-30T06:46:15.195ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59292025-04-30T06:46:15.195ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59302025-04-30T06:46:15.195ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59312025-04-30T06:46:15.195ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59322025-04-30T06:46:15.206ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59332025-04-30T06:46:15.210ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59342025-04-30T06:46:15.210ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59352025-04-30T06:46:15.211ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59362025-04-30T06:46:15.211ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59372025-04-30T06:46:15.211ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59382025-04-30T06:46:15.211ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59392025-04-30T06:46:15.225ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59402025-04-30T06:46:15.230ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59412025-04-30T06:46:15.230ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59422025-04-30T06:46:15.230ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59432025-04-30T06:46:15.230ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59442025-04-30T06:46:15.233ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
59452025-04-30T06:46:15.281ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
59462025-04-30T06:46:15.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
59472025-04-30T06:46:15.284ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59482025-04-30T06:46:15.284ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59492025-04-30T06:46:15.284ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59502025-04-30T06:46:15.284ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59512025-04-30T06:46:15.285ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59522025-04-30T06:46:15.285ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:50433 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59532025-04-30T06:46:15.286ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:50433 req_id = af43f1b7-9da7-46e0-901b-4e232ce11ba3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59542025-04-30T06:46:15.286ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 525 local_addr = [::1]:54590 method = GET remote_addr = [::1]:50433 req_id = af43f1b7-9da7-46e0-901b-4e232ce11ba3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59552025-04-30T06:46:15.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "af43f1b7-9da7-46e0-901b-4e232ce11ba3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
59562025-04-30T06:46:15.288ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59572025-04-30T06:46:15.288ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59582025-04-30T06:46:15.288ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
59592025-04-30T06:46:15.291ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59602025-04-30T06:46:15.291ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59612025-04-30T06:46:15.291ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59622025-04-30T06:46:15.291ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59632025-04-30T06:46:15.291ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59642025-04-30T06:46:15.292ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59652025-04-30T06:46:15.297ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59662025-04-30T06:46:15.297ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59672025-04-30T06:46:15.297ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59682025-04-30T06:46:15.300ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59692025-04-30T06:46:15.302ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59702025-04-30T06:46:15.316ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59712025-04-30T06:46:15.316ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59722025-04-30T06:46:15.316ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59732025-04-30T06:46:15.317ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59742025-04-30T06:46:15.317ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59752025-04-30T06:46:15.317ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
59762025-04-30T06:46:15.318ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59772025-04-30T06:46:15.401ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
59782025-04-30T06:46:15.425ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
59792025-04-30T06:46:15.425ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
59802025-04-30T06:46:15.426ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 700.160834ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
59812025-04-30T06:46:15.426ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
59822025-04-30T06:46:15.426ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59832025-04-30T06:46:15.426ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59842025-04-30T06:46:15.426ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59852025-04-30T06:46:15.427ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59862025-04-30T06:46:15.428ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59872025-04-30T06:46:15.440ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59882025-04-30T06:46:15.444ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59892025-04-30T06:46:15.444ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59902025-04-30T06:46:15.444ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59912025-04-30T06:46:15.445ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59922025-04-30T06:46:15.450ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
59932025-04-30T06:46:15.450ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
59942025-04-30T06:46:15.452ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 09 56 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .V..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59544 req_id = 729be951-f56a-435a-9fd0-9c4ba8cbfef3
59952025-04-30T06:46:15.452ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2390,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59544 req_id = 729be951-f56a-435a-9fd0-9c4ba8cbfef3
59962025-04-30T06:46:15.452ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
59972025-04-30T06:46:15.452ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59982025-04-30T06:46:15.452ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59544 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 729be951-f56a-435a-9fd0-9c4ba8cbfef3
59992025-04-30T06:46:15.452ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60002025-04-30T06:46:15.452ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60012025-04-30T06:46:15.452ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60022025-04-30T06:46:15.452ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
60032025-04-30T06:46:15.453ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60042025-04-30T06:46:15.453ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4440, tv_nsec: 252469154 } })
60052025-04-30T06:46:15.453ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
60062025-04-30T06:46:15.454ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: dd 63 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .c..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55221 req_id = eac71bfc-8558-4d1a-bbf7-91c41e571d79
60072025-04-30T06:46:15.454ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56675,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55221 req_id = eac71bfc-8558-4d1a-bbf7-91c41e571d79
60082025-04-30T06:46:15.454ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60092025-04-30T06:46:15.454ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55221 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = eac71bfc-8558-4d1a-bbf7-91c41e571d79
60102025-04-30T06:46:15.498ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60112025-04-30T06:46:15.500ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 75 92 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c u............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65233 req_id = 73128555-9505-4fe1-b4c0-e537ffd92b37
60122025-04-30T06:46:15.500ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30098,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65233 req_id = 73128555-9505-4fe1-b4c0-e537ffd92b37
60132025-04-30T06:46:15.500ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
60142025-04-30T06:46:15.500ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30098, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65233 req_id = 73128555-9505-4fe1-b4c0-e537ffd92b37
60152025-04-30T06:46:15.500ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60162025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60172025-04-30T06:46:15.506ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60182025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
60192025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60202025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
60212025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
60222025-04-30T06:46:15.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
60232025-04-30T06:46:15.509ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2046126a-96b8-4eca-82eb-dc86b6b99b19", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60242025-04-30T06:46:15.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
60252025-04-30T06:46:15.516ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60262025-04-30T06:46:15.516ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
60272025-04-30T06:46:15.517ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60282025-04-30T06:46:15.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ce07841-a971-4083-aef1-a3accec01fe0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60292025-04-30T06:46:15.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
60302025-04-30T06:46:15.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9775cbb-b62c-4fdb-9d32-64ae8f8813e5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60312025-04-30T06:46:15.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
60322025-04-30T06:46:15.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86fbffe5-62ca-4242-aeee-70fb40228d18", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60332025-04-30T06:46:15.524ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
60342025-04-30T06:46:15.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "992b3e87-038c-4c5e-a6d3-ba5f64aff228", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60352025-04-30T06:46:15.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
60362025-04-30T06:46:15.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b826e70d-f53b-4231-8051-08a075e01041", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60372025-04-30T06:46:15.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
60382025-04-30T06:46:15.532ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5caecc7-5c53-4576-96cd-f064c19b37af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60392025-04-30T06:46:15.532ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
60402025-04-30T06:46:15.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0526e5d0-a314-47cf-a6e4-ba4c3858ba7f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60412025-04-30T06:46:15.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
60422025-04-30T06:46:15.537ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fd6ff71-a46a-45b8-a747-8381686752fb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60432025-04-30T06:46:15.537ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
60442025-04-30T06:46:15.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee33d613-c295-478c-9c68-0e0cc5dc28bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60452025-04-30T06:46:15.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
60462025-04-30T06:46:15.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "272adb3c-531b-4fbd-825a-4e07725cde24", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60472025-04-30T06:46:15.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
60482025-04-30T06:46:15.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c0e198-a1d9-4f55-a614-33ebfcf7795c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60492025-04-30T06:46:15.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
60502025-04-30T06:46:15.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f19dbcc0-17ed-430b-b959-34e384d55d8d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60512025-04-30T06:46:15.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
60522025-04-30T06:46:15.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5960bfca-04b8-4c45-98fc-39e04a994b72", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60532025-04-30T06:46:15.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
60542025-04-30T06:46:15.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19022b0-ba8b-428f-bf6c-046c7a57e074", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60552025-04-30T06:46:15.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
60562025-04-30T06:46:15.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
60572025-04-30T06:46:15.548ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60582025-04-30T06:46:15.549ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 38e3724b-d053-4060-9239-7f9d973d53d4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60592025-04-30T06:46:15.549ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 38e3724b-d053-4060-9239-7f9d973d53d4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60602025-04-30T06:46:15.549ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 38e3724b-d053-4060-9239-7f9d973d53d4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60612025-04-30T06:46:15.550ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 38e3724b-d053-4060-9239-7f9d973d53d4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60622025-04-30T06:46:15.553ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 4084 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 38e3724b-d053-4060-9239-7f9d973d53d4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60632025-04-30T06:46:15.553ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "38e3724b-d053-4060-9239-7f9d973d53d4", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60642025-04-30T06:46:15.554ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60652025-04-30T06:46:15.554ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 87778ebf-4024-4b15-bf0c-0efb426ca79c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60662025-04-30T06:46:15.555ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 87778ebf-4024-4b15-bf0c-0efb426ca79c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60672025-04-30T06:46:15.556ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 87778ebf-4024-4b15-bf0c-0efb426ca79c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60682025-04-30T06:46:15.557ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 87778ebf-4024-4b15-bf0c-0efb426ca79c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60692025-04-30T06:46:15.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fff4170-184c-45d0-a9aa-cd4d8b44d2e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60702025-04-30T06:46:15.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
60712025-04-30T06:46:15.560ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6107 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 87778ebf-4024-4b15-bf0c-0efb426ca79c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60722025-04-30T06:46:15.561ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "87778ebf-4024-4b15-bf0c-0efb426ca79c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60732025-04-30T06:46:15.561ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
60742025-04-30T06:46:15.561ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60752025-04-30T06:46:15.561ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
60762025-04-30T06:46:15.562ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c02521-d4b2-4225-82de-1ea6442df31c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60772025-04-30T06:46:15.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
60782025-04-30T06:46:15.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7012b12-c71e-44d2-a812-ddc46ab0ed1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60792025-04-30T06:46:15.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
60802025-04-30T06:46:15.567ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
60812025-04-30T06:46:15.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a637bc-f61a-4317-97cb-0ea334fa99b2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60822025-04-30T06:46:15.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
60832025-04-30T06:46:15.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8eab18a-93ac-4eb5-acb0-cdd9b8da82b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60842025-04-30T06:46:15.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
60852025-04-30T06:46:15.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96643347-bfec-474b-9db8-58de60eca66c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60862025-04-30T06:46:15.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
60872025-04-30T06:46:15.573ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad8234b5-d483-43e9-a6ed-a3a30a09d268", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60882025-04-30T06:46:15.573ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
60892025-04-30T06:46:15.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f80521b-78fe-4c1b-a8b4-913f9151b577", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60902025-04-30T06:46:15.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
60912025-04-30T06:46:15.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f58b80e-8930-48dd-9dd6-88c7215bce62", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60922025-04-30T06:46:15.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
60932025-04-30T06:46:15.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b97022-82ed-480f-8373-15920c4f147f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60942025-04-30T06:46:15.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
60952025-04-30T06:46:15.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a1a5f08-1599-4802-acc1-368996387548", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60962025-04-30T06:46:15.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
60972025-04-30T06:46:15.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aff211c-5ef4-4d97-a0d9-486cb5b8a228", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
60982025-04-30T06:46:15.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
60992025-04-30T06:46:15.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c660f937-38bf-4975-9da5-859236fc9c43", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61002025-04-30T06:46:15.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
61012025-04-30T06:46:15.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64871f7c-5be8-4318-842b-8ea4d3657368", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61022025-04-30T06:46:15.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
61032025-04-30T06:46:15.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "318e79ec-4c0b-43e9-a260-6b6bfd9a338d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61042025-04-30T06:46:15.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
61052025-04-30T06:46:15.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fd77be6-fcfa-43b6-981b-1ef395d3ada5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61062025-04-30T06:46:15.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
61072025-04-30T06:46:15.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb04bc4d-391b-44ab-b0de-ebfa94b1ab6b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61082025-04-30T06:46:15.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
61092025-04-30T06:46:15.590ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7750d7a-a413-4575-9d8a-3b0069250ec8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61102025-04-30T06:46:15.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
61112025-04-30T06:46:15.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d61af92-afd8-4e33-aee9-eccadef3a4e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61122025-04-30T06:46:15.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
61132025-04-30T06:46:15.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f33b3775-1ea3-40c8-b786-60c8cf6ae1f7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61142025-04-30T06:46:15.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
61152025-04-30T06:46:15.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db059b88-ecb0-4efc-a638-3960a118c2d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61162025-04-30T06:46:15.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
61172025-04-30T06:46:15.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "487b611a-ad01-48de-9444-0270bdee3706", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61182025-04-30T06:46:15.599ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
61192025-04-30T06:46:15.600ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "471262b1-d61f-41c7-8dc2-e18d1897db40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61202025-04-30T06:46:15.600ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
61212025-04-30T06:46:15.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "899f243d-6292-4c2a-b071-17e922357e68", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61222025-04-30T06:46:15.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
61232025-04-30T06:46:15.605ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f52beb97-10ac-494f-862e-760354609919", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61242025-04-30T06:46:15.605ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
61252025-04-30T06:46:15.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfba21db-192f-473d-9463-b2ee4a9f7b70", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61262025-04-30T06:46:15.608ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
61272025-04-30T06:46:15.609ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "088119a8-53eb-4851-a8b6-680923a0194e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61282025-04-30T06:46:15.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
61292025-04-30T06:46:15.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1f50b4f-4615-47af-aadf-f48c5528dd93", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61302025-04-30T06:46:15.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
61312025-04-30T06:46:15.611ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de4990a6-dd38-4108-ade5-2994cbbf0db4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61322025-04-30T06:46:15.612ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
61332025-04-30T06:46:15.613ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad5e9ba-80d9-465c-ab82-5d9e7a003ccf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:14 GMT"} })
61342025-04-30T06:46:15.614ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61352025-04-30T06:46:15.614ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61362025-04-30T06:46:15.614ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61372025-04-30T06:46:15.614ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61382025-04-30T06:46:15.614ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61392025-04-30T06:46:15.617ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61402025-04-30T06:46:15.622ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61412025-04-30T06:46:15.622ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61422025-04-30T06:46:15.622ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61432025-04-30T06:46:15.622ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61442025-04-30T06:46:15.622ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61452025-04-30T06:46:15.623ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61462025-04-30T06:46:15.623ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61472025-04-30T06:46:15.623ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61482025-04-30T06:46:15.623ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61492025-04-30T06:46:15.623ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
61502025-04-30T06:46:15.623ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61512025-04-30T06:46:15.626ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61522025-04-30T06:46:15.631ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61532025-04-30T06:46:15.631ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61542025-04-30T06:46:15.631ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61552025-04-30T06:46:15.631ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61562025-04-30T06:46:15.633ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61572025-04-30T06:46:15.646ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61582025-04-30T06:46:15.646ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61592025-04-30T06:46:15.646ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
61602025-04-30T06:46:15.647ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61612025-04-30T06:46:15.695ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
61622025-04-30T06:46:15.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
61632025-04-30T06:46:15.697ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:64870 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61642025-04-30T06:46:15.697ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61652025-04-30T06:46:15.698ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61662025-04-30T06:46:15.698ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:64870 req_id = 074f4271-a23a-4fb8-821e-62ae87413586 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61672025-04-30T06:46:15.700ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1603 local_addr = [::1]:54590 method = GET remote_addr = [::1]:64870 req_id = 074f4271-a23a-4fb8-821e-62ae87413586 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61682025-04-30T06:46:15.700ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61692025-04-30T06:46:15.700ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
61702025-04-30T06:46:15.702ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61712025-04-30T06:46:15.703ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61722025-04-30T06:46:15.704ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61732025-04-30T06:46:15.704ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61742025-04-30T06:46:15.704ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61752025-04-30T06:46:15.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "074f4271-a23a-4fb8-821e-62ae87413586", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
61762025-04-30T06:46:15.707ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61772025-04-30T06:46:15.707ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61782025-04-30T06:46:15.707ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
61792025-04-30T06:46:15.710ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61802025-04-30T06:46:15.710ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61812025-04-30T06:46:15.710ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61822025-04-30T06:46:15.710ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61832025-04-30T06:46:15.711ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61842025-04-30T06:46:15.715ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61852025-04-30T06:46:15.721ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
61862025-04-30T06:46:15.722ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61872025-04-30T06:46:15.722ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61882025-04-30T06:46:15.722ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61892025-04-30T06:46:15.773ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
61902025-04-30T06:46:15.776ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
61912025-04-30T06:46:15.776ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
61922025-04-30T06:46:15.776ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 350.297986ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
61932025-04-30T06:46:15.776ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
61942025-04-30T06:46:15.776ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61952025-04-30T06:46:15.776ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61962025-04-30T06:46:15.776ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61972025-04-30T06:46:15.776ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61982025-04-30T06:46:15.777ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61992025-04-30T06:46:15.780ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62002025-04-30T06:46:15.785ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62012025-04-30T06:46:15.785ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62022025-04-30T06:46:15.785ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62032025-04-30T06:46:15.786ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62042025-04-30T06:46:15.789ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
62052025-04-30T06:46:15.790ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62062025-04-30T06:46:15.790ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 23 8e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 #............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52651 req_id = ce598aec-79c4-4c4b-9332-647ecda9768c
62072025-04-30T06:46:15.790ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9102,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52651 req_id = ce598aec-79c4-4c4b-9332-647ecda9768c
62082025-04-30T06:46:15.790ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
62092025-04-30T06:46:15.791ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62102025-04-30T06:46:15.791ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52651 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ce598aec-79c4-4c4b-9332-647ecda9768c
62112025-04-30T06:46:15.792ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4440, tv_nsec: 591195604 } })
62122025-04-30T06:46:15.792ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
62132025-04-30T06:46:15.792ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 31 50 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 1P..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62654 req_id = b8dccff3-4493-47cd-9b7b-52f80bcba9ec
62142025-04-30T06:46:15.792ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12624,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62654 req_id = b8dccff3-4493-47cd-9b7b-52f80bcba9ec
62152025-04-30T06:46:15.792ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62162025-04-30T06:46:15.792ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62654 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b8dccff3-4493-47cd-9b7b-52f80bcba9ec
62172025-04-30T06:46:15.835ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62182025-04-30T06:46:15.836ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62192025-04-30T06:46:15.836ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62202025-04-30T06:46:15.836ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62212025-04-30T06:46:15.836ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
62222025-04-30T06:46:15.837ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62232025-04-30T06:46:15.838ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e8 6c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .l..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49917 req_id = 74fbd72c-7dab-429b-b44b-4e7d194c749a
62242025-04-30T06:46:15.838ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59500,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49917 req_id = 74fbd72c-7dab-429b-b44b-4e7d194c749a
62252025-04-30T06:46:15.838ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
62262025-04-30T06:46:15.838ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59500, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49917 req_id = 74fbd72c-7dab-429b-b44b-4e7d194c749a
62272025-04-30T06:46:15.838ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
62282025-04-30T06:46:15.839ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
62292025-04-30T06:46:15.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
62302025-04-30T06:46:15.840ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62312025-04-30T06:46:15.845ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62322025-04-30T06:46:15.845ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62332025-04-30T06:46:15.846ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
62342025-04-30T06:46:15.846ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62352025-04-30T06:46:15.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c70f3229-d7b0-4641-bfa7-c86115958a1b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62362025-04-30T06:46:15.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
62372025-04-30T06:46:15.849ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62382025-04-30T06:46:15.849ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
62392025-04-30T06:46:15.850ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62402025-04-30T06:46:15.850ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d5df2a4-d3d8-430d-92d2-e5992d9a69ac", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62412025-04-30T06:46:15.850ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
62422025-04-30T06:46:15.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e34759c-bd85-48db-b236-cce7a18072fd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62432025-04-30T06:46:15.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
62442025-04-30T06:46:15.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "830f57fd-08dd-4b4b-8cd1-f268c35c9dca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62452025-04-30T06:46:15.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
62462025-04-30T06:46:15.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2869e9c-8df8-4196-9379-c9f4df18d486", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62472025-04-30T06:46:15.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
62482025-04-30T06:46:15.861ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b98acb76-430b-4bc7-91aa-2d2a0f4a1c00", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62492025-04-30T06:46:15.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
62502025-04-30T06:46:15.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bbad2e1-d5b0-4f09-b0b4-2e9acd5c2dcf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62512025-04-30T06:46:15.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
62522025-04-30T06:46:15.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc2ad72-b0eb-4e91-86d3-0a433089ea13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62532025-04-30T06:46:15.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
62542025-04-30T06:46:15.868ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9abdb9-e22f-43bf-ab81-4ebacb2bff4f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62552025-04-30T06:46:15.868ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
62562025-04-30T06:46:15.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb1d47f3-3a93-445f-8079-e7bf3a4193aa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62572025-04-30T06:46:15.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
62582025-04-30T06:46:15.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1045f7e2-ebdf-4360-84be-6e4e9dbd82ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62592025-04-30T06:46:15.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
62602025-04-30T06:46:15.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccbe5c6b-46f4-47f6-836f-2f31a7651192", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62612025-04-30T06:46:15.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
62622025-04-30T06:46:15.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1677d9a-417a-41f0-9448-cf50ce1d5e52", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62632025-04-30T06:46:15.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
62642025-04-30T06:46:15.889ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
62652025-04-30T06:46:15.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29582586-ee38-44ca-89c4-b475807b6000", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62662025-04-30T06:46:15.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
62672025-04-30T06:46:15.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e797d261-bfd6-408f-9efe-5c688ff10027", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62682025-04-30T06:46:15.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
62692025-04-30T06:46:15.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cad8140-aba8-4570-b5e9-a4720f79518b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62702025-04-30T06:46:15.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
62712025-04-30T06:46:15.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27cfcab2-b47c-440f-8b0b-8eb690e8c42d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62722025-04-30T06:46:15.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
62732025-04-30T06:46:15.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93a18fa5-0c63-43bf-b2d9-39f0162fac54", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62742025-04-30T06:46:15.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
62752025-04-30T06:46:15.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c85c382-aea4-44d8-9303-7d56789a7e5b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62762025-04-30T06:46:15.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
62772025-04-30T06:46:15.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e336174-6da1-4330-b119-2c64aad75618", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62782025-04-30T06:46:15.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
62792025-04-30T06:46:15.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24b3287b-52da-4241-8768-bcedc82338ed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62802025-04-30T06:46:15.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
62812025-04-30T06:46:15.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c8f196d-0f21-4d02-a6f1-b5fd4fdb3257", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62822025-04-30T06:46:15.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
62832025-04-30T06:46:15.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a48eacd-a19f-4c31-89fa-904487df4ebf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62842025-04-30T06:46:15.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
62852025-04-30T06:46:15.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "194447ad-e8db-49da-a3c1-09b83bc51baf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62862025-04-30T06:46:15.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
62872025-04-30T06:46:15.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36631d3d-52e0-4a52-8bc8-02086127b949", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62882025-04-30T06:46:15.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
62892025-04-30T06:46:15.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a31863d3-00df-467a-8a7f-64fc343efb4b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62902025-04-30T06:46:15.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
62912025-04-30T06:46:15.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad6dd11b-f2b7-4d12-93f9-9a4d22f1568b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62922025-04-30T06:46:15.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
62932025-04-30T06:46:15.925ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22cd4918-d6aa-4794-8129-aa362332e923", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62942025-04-30T06:46:15.925ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
62952025-04-30T06:46:15.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ad5b0bc-a784-4381-818a-21632a8b7d0a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62962025-04-30T06:46:15.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
62972025-04-30T06:46:15.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ecf506f-13c4-4790-beff-df699071e405", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
62982025-04-30T06:46:15.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
62992025-04-30T06:46:15.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ed8cc99-c142-445a-8c01-d75c6179cab9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63002025-04-30T06:46:15.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
63012025-04-30T06:46:15.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f259e9-2581-4174-89e0-3e11364ffea5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63022025-04-30T06:46:15.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
63032025-04-30T06:46:15.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86fbc4ae-ec02-44d9-b198-f76e2eeccf07", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63042025-04-30T06:46:15.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
63052025-04-30T06:46:15.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1e1b654-904a-48a4-9750-56e68613e788", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63062025-04-30T06:46:15.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
63072025-04-30T06:46:15.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7baaa6f-7052-4ef1-8f0d-1c566320962a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63082025-04-30T06:46:15.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
63092025-04-30T06:46:15.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3afa2227-c779-4563-8493-e77a46128b4d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63102025-04-30T06:46:15.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
63112025-04-30T06:46:15.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1748f6b-80ff-4e33-837f-b03738937e3c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63122025-04-30T06:46:15.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
63132025-04-30T06:46:15.940ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63142025-04-30T06:46:15.940ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63152025-04-30T06:46:15.940ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63162025-04-30T06:46:15.940ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63172025-04-30T06:46:15.941ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63182025-04-30T06:46:15.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "577ca1bc-c581-4e42-b712-a3ef41daa563", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63192025-04-30T06:46:15.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
63202025-04-30T06:46:15.944ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63212025-04-30T06:46:15.949ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63222025-04-30T06:46:15.949ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63232025-04-30T06:46:15.949ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63242025-04-30T06:46:15.949ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63252025-04-30T06:46:15.952ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63262025-04-30T06:46:15.952ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63272025-04-30T06:46:15.953ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63282025-04-30T06:46:15.955ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52578cd9-596a-4d11-b4ca-55d40206b19e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63292025-04-30T06:46:15.955ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
63302025-04-30T06:46:15.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "241977e2-a4c3-4895-9b74-9c83b892b6df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63312025-04-30T06:46:15.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
63322025-04-30T06:46:15.964ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfed826f-1cd0-460b-951b-385107510a98", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63332025-04-30T06:46:15.970ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
63342025-04-30T06:46:15.973ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1c4894-f24c-4f47-8888-475ad8e21474", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63352025-04-30T06:46:15.974ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
63362025-04-30T06:46:15.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec0937b-0db8-4143-a9d5-62bb37df9461", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63372025-04-30T06:46:15.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
63382025-04-30T06:46:15.988ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63392025-04-30T06:46:15.988ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63402025-04-30T06:46:15.991ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
63412025-04-30T06:46:15.993ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
63422025-04-30T06:46:15.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84e5e1d1-80f4-4fd1-a57c-0f8b83542cb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63432025-04-30T06:46:15.998ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
63442025-04-30T06:46:15.998ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
63452025-04-30T06:46:15.998ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
63462025-04-30T06:46:16.008ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
63472025-04-30T06:46:16.008ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
63482025-04-30T06:46:16.012ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed75fc08-3a30-4a52-9828-67a12900fc64", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63492025-04-30T06:46:16.012ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63502025-04-30T06:46:16.012ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63512025-04-30T06:46:16.012ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63522025-04-30T06:46:16.012ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63532025-04-30T06:46:16.013ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63542025-04-30T06:46:16.015ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
63552025-04-30T06:46:16.054ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63562025-04-30T06:46:16.065ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63572025-04-30T06:46:16.065ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63582025-04-30T06:46:16.065ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63592025-04-30T06:46:16.065ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63602025-04-30T06:46:16.066ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63612025-04-30T06:46:16.071ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63622025-04-30T06:46:16.073ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63632025-04-30T06:46:16.076ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63642025-04-30T06:46:16.076ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63652025-04-30T06:46:16.076ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63662025-04-30T06:46:16.076ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63672025-04-30T06:46:16.081ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63682025-04-30T06:46:16.087ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63692025-04-30T06:46:16.087ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63702025-04-30T06:46:16.087ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63712025-04-30T06:46:16.088ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63722025-04-30T06:46:16.114ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63732025-04-30T06:46:16.122ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63742025-04-30T06:46:16.122ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63752025-04-30T06:46:16.122ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63762025-04-30T06:46:16.122ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63772025-04-30T06:46:16.123ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
63782025-04-30T06:46:16.223ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
63792025-04-30T06:46:16.223ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
63802025-04-30T06:46:16.227ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:61687 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
63812025-04-30T06:46:16.228ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63822025-04-30T06:46:16.228ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
63832025-04-30T06:46:16.229ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63842025-04-30T06:46:16.229ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:61687 req_id = a141b73b-f132-47fe-9c31-bed529d988a6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63852025-04-30T06:46:16.231ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1373 local_addr = [::1]:54590 method = GET remote_addr = [::1]:61687 req_id = a141b73b-f132-47fe-9c31-bed529d988a6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63862025-04-30T06:46:16.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a141b73b-f132-47fe-9c31-bed529d988a6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
63872025-04-30T06:46:16.234ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63882025-04-30T06:46:16.234ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63892025-04-30T06:46:16.234ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
63902025-04-30T06:46:16.236ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
63912025-04-30T06:46:16.236ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63922025-04-30T06:46:16.236ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63932025-04-30T06:46:16.236ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63942025-04-30T06:46:16.236ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63952025-04-30T06:46:16.245ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63962025-04-30T06:46:16.250ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
63972025-04-30T06:46:16.250ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63982025-04-30T06:46:16.250ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63992025-04-30T06:46:16.251ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64002025-04-30T06:46:16.289ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
64012025-04-30T06:46:16.295ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64022025-04-30T06:46:16.295ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64032025-04-30T06:46:16.295ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 518.780937ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64042025-04-30T06:46:16.295ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64052025-04-30T06:46:16.295ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64062025-04-30T06:46:16.295ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64072025-04-30T06:46:16.295ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64082025-04-30T06:46:16.295ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64092025-04-30T06:46:16.296ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64102025-04-30T06:46:16.304ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64112025-04-30T06:46:16.309ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64122025-04-30T06:46:16.310ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64132025-04-30T06:46:16.310ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64142025-04-30T06:46:16.310ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64152025-04-30T06:46:16.313ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
64162025-04-30T06:46:16.313ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64172025-04-30T06:46:16.314ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 98 f7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63115 req_id = 574c0ba3-bbd4-454b-a19b-cccb64bdb0cd
64182025-04-30T06:46:16.314ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39159,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63115 req_id = 574c0ba3-bbd4-454b-a19b-cccb64bdb0cd
64192025-04-30T06:46:16.314ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
64202025-04-30T06:46:16.314ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64212025-04-30T06:46:16.314ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63115 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 574c0ba3-bbd4-454b-a19b-cccb64bdb0cd
64222025-04-30T06:46:16.315ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4441, tv_nsec: 114228744 } })
64232025-04-30T06:46:16.315ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
64242025-04-30T06:46:16.315ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d 51 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }Q..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50175 req_id = 54631e54-bfcd-4323-ba5f-cfcdb065da7e
64252025-04-30T06:46:16.315ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32081,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50175 req_id = 54631e54-bfcd-4323-ba5f-cfcdb065da7e
64262025-04-30T06:46:16.315ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64272025-04-30T06:46:16.316ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50175 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 54631e54-bfcd-4323-ba5f-cfcdb065da7e
64282025-04-30T06:46:16.364ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64292025-04-30T06:46:16.365ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64302025-04-30T06:46:16.365ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64312025-04-30T06:46:16.365ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64322025-04-30T06:46:16.366ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
64332025-04-30T06:46:16.366ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64342025-04-30T06:46:16.367ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 47 bf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c G............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59993 req_id = 8ba59d29-860b-4bb0-bfe5-bd4d141bd6a7
64352025-04-30T06:46:16.367ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18367,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59993 req_id = 8ba59d29-860b-4bb0-bfe5-bd4d141bd6a7
64362025-04-30T06:46:16.367ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
64372025-04-30T06:46:16.367ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18367, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59993 req_id = 8ba59d29-860b-4bb0-bfe5-bd4d141bd6a7
64382025-04-30T06:46:16.367ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
64392025-04-30T06:46:16.367ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
64402025-04-30T06:46:16.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
64412025-04-30T06:46:16.369ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64422025-04-30T06:46:16.375ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64432025-04-30T06:46:16.375ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64442025-04-30T06:46:16.375ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
64452025-04-30T06:46:16.376ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64462025-04-30T06:46:16.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc361e8-ff2d-4324-9dc2-481b8c20c713", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64472025-04-30T06:46:16.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
64482025-04-30T06:46:16.381ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64492025-04-30T06:46:16.381ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
64502025-04-30T06:46:16.382ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64512025-04-30T06:46:16.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "265d1fbb-860e-4687-b45a-0e53d691a4b3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64522025-04-30T06:46:16.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
64532025-04-30T06:46:16.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e15c81e4-e055-4c1f-b167-53cb08688929", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64542025-04-30T06:46:16.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
64552025-04-30T06:46:16.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74dcb26e-05b4-42cc-a938-afbd98205042", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64562025-04-30T06:46:16.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
64572025-04-30T06:46:16.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f316e29b-5262-4121-9973-0807c1906faf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64582025-04-30T06:46:16.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
64592025-04-30T06:46:16.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9facb887-4c9d-4fb4-881b-b8f3047c58bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64602025-04-30T06:46:16.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
64612025-04-30T06:46:16.424ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
64622025-04-30T06:46:16.446ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f5c0f81-7926-46f1-a57a-557bc7855a76", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64632025-04-30T06:46:16.446ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
64642025-04-30T06:46:16.449ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ab797b-cd18-4511-b48c-eb4cb515d51e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64652025-04-30T06:46:16.449ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
64662025-04-30T06:46:16.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1efd5214-b268-4ab9-953a-f89210178c80", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64672025-04-30T06:46:16.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
64682025-04-30T06:46:16.451ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c5abee2-dfc9-4e17-a5d8-4af48441a638", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64692025-04-30T06:46:16.451ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
64702025-04-30T06:46:16.453ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9db7f43e-443c-47c4-b5c5-57b575e19404", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64712025-04-30T06:46:16.453ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
64722025-04-30T06:46:16.454ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4293575f-9aa1-46c0-b697-245f504937d9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64732025-04-30T06:46:16.454ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
64742025-04-30T06:46:16.455ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c551cd-c071-4eae-a57b-149948afe5fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64752025-04-30T06:46:16.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
64762025-04-30T06:46:16.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c91214f7-4579-406e-83ac-3fe71db3ebee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64772025-04-30T06:46:16.458ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
64782025-04-30T06:46:16.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16417fe5-2845-4b69-9615-943bdb433663", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64792025-04-30T06:46:16.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
64802025-04-30T06:46:16.461ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f26b74-ec04-41a9-a86c-7069d2ee85fe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64812025-04-30T06:46:16.461ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
64822025-04-30T06:46:16.463ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa80925e-bfc7-499e-b4fd-46f46373dff9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64832025-04-30T06:46:16.463ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
64842025-04-30T06:46:16.465ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31cef7e0-6836-4148-a504-bd6d6e34e450", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64852025-04-30T06:46:16.465ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
64862025-04-30T06:46:16.467ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68f9a0f-d9aa-4acc-b02c-8da81048fc24", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64872025-04-30T06:46:16.467ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
64882025-04-30T06:46:16.468ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fe6ca6a-6f3a-45f5-acaf-30d209ed1fb1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64892025-04-30T06:46:16.469ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
64902025-04-30T06:46:16.469ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64e5d48c-ecf9-4f66-9c52-4146f30402c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64912025-04-30T06:46:16.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
64922025-04-30T06:46:16.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f181d141-1730-4d21-a674-53ef70afe23b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64932025-04-30T06:46:16.471ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
64942025-04-30T06:46:16.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27630063-7acd-4b82-aac0-79c3f7c9089e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64952025-04-30T06:46:16.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
64962025-04-30T06:46:16.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca26c2c-9e8f-461b-a791-860cfc288926", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
64972025-04-30T06:46:16.475ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
64982025-04-30T06:46:16.475ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64992025-04-30T06:46:16.475ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65002025-04-30T06:46:16.475ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65012025-04-30T06:46:16.475ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
65022025-04-30T06:46:16.476ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65032025-04-30T06:46:16.479ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "844f8565-5bc9-40f6-94a3-d97e7e1c29b6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65042025-04-30T06:46:16.479ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
65052025-04-30T06:46:16.479ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65062025-04-30T06:46:16.486ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65072025-04-30T06:46:16.486ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65082025-04-30T06:46:16.486ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
65092025-04-30T06:46:16.486ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65102025-04-30T06:46:16.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "305abd6a-319c-43f7-8396-e6e271a0bae0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65112025-04-30T06:46:16.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
65122025-04-30T06:46:16.489ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65132025-04-30T06:46:16.489ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
65142025-04-30T06:46:16.490ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65152025-04-30T06:46:16.492ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0586e053-8d74-4914-95b7-fb87818d4ff3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65162025-04-30T06:46:16.492ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
65172025-04-30T06:46:16.494ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c64d4f9-3288-44af-b229-c8dea4e294ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65182025-04-30T06:46:16.494ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
65192025-04-30T06:46:16.496ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d3d8e7-6b15-4229-8b27-2948174cb137", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65202025-04-30T06:46:16.496ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
65212025-04-30T06:46:16.499ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa188d1c-c178-45d5-b58d-02a4474b5ae1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65222025-04-30T06:46:16.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
65232025-04-30T06:46:16.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0fa4e84-aa89-46f0-85a0-ea87c55181da", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65242025-04-30T06:46:16.501ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
65252025-04-30T06:46:16.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "816e6b8a-a650-40da-a0c9-a52e7e62d4ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65262025-04-30T06:46:16.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
65272025-04-30T06:46:16.505ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bbada42-8297-4740-9177-ea677ff9a89b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65282025-04-30T06:46:16.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
65292025-04-30T06:46:16.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f24a21a-ed20-41c5-aea0-9478541bcb20", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65302025-04-30T06:46:16.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
65312025-04-30T06:46:16.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "527765b7-2858-4860-995f-0987da51ff88", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65322025-04-30T06:46:16.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
65332025-04-30T06:46:16.511ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b914b69f-270c-49f5-aae7-9305023ed3c1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65342025-04-30T06:46:16.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
65352025-04-30T06:46:16.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ee2ddb-2200-407e-8c34-41876d644ecd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65362025-04-30T06:46:16.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
65372025-04-30T06:46:16.516ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e9f97c9-e605-4ca2-9a36-26f7a9b25eb6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65382025-04-30T06:46:16.516ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
65392025-04-30T06:46:16.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde48cf7-9769-4564-9ee9-96121a610c0c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65402025-04-30T06:46:16.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
65412025-04-30T06:46:16.520ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
65422025-04-30T06:46:16.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1155206f-eebc-401d-8e3f-040199a83979", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65432025-04-30T06:46:16.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
65442025-04-30T06:46:16.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66a9a8e9-6e54-4994-970b-930960e2cd15", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65452025-04-30T06:46:16.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
65462025-04-30T06:46:16.524ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9920458d-4f21-4d5a-b92f-5345574c2333", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65472025-04-30T06:46:16.524ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
65482025-04-30T06:46:16.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a93d4b7-8a94-4b07-bfb8-a5a6b3b86e7f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65492025-04-30T06:46:16.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
65502025-04-30T06:46:16.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d7e53e-f756-4fe5-8813-bfa78d9c0284", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65512025-04-30T06:46:16.527ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
65522025-04-30T06:46:16.528ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d0e2b9c-6da7-4dbc-b0a9-a822e5b424ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65532025-04-30T06:46:16.528ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65542025-04-30T06:46:16.528ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65552025-04-30T06:46:16.528ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65562025-04-30T06:46:16.528ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65572025-04-30T06:46:16.529ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65582025-04-30T06:46:16.531ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65592025-04-30T06:46:16.536ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65602025-04-30T06:46:16.536ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65612025-04-30T06:46:16.536ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65622025-04-30T06:46:16.536ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65632025-04-30T06:46:16.536ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65642025-04-30T06:46:16.537ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65652025-04-30T06:46:16.539ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65662025-04-30T06:46:16.543ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65672025-04-30T06:46:16.543ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65682025-04-30T06:46:16.543ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65692025-04-30T06:46:16.543ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65702025-04-30T06:46:16.592ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
65712025-04-30T06:46:16.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
65722025-04-30T06:46:16.592ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65732025-04-30T06:46:16.592ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65742025-04-30T06:46:16.592ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65752025-04-30T06:46:16.593ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
65762025-04-30T06:46:16.593ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65772025-04-30T06:46:16.593ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:51235 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
65782025-04-30T06:46:16.594ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:51235 req_id = 0d717aed-ca31-4b78-b40f-0ac23b90cdef sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65792025-04-30T06:46:16.595ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 491 local_addr = [::1]:54590 method = GET remote_addr = [::1]:51235 req_id = 0d717aed-ca31-4b78-b40f-0ac23b90cdef response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65802025-04-30T06:46:16.595ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d717aed-ca31-4b78-b40f-0ac23b90cdef", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:15 GMT"} })
65812025-04-30T06:46:16.597ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65822025-04-30T06:46:16.597ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65832025-04-30T06:46:16.597ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
65842025-04-30T06:46:16.599ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65852025-04-30T06:46:16.599ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65862025-04-30T06:46:16.599ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65872025-04-30T06:46:16.599ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65882025-04-30T06:46:16.600ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65892025-04-30T06:46:16.600ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65902025-04-30T06:46:16.606ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65912025-04-30T06:46:16.606ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65922025-04-30T06:46:16.606ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
65932025-04-30T06:46:16.606ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65942025-04-30T06:46:16.608ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65952025-04-30T06:46:16.612ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65962025-04-30T06:46:16.612ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65972025-04-30T06:46:16.612ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65982025-04-30T06:46:16.613ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65992025-04-30T06:46:16.623ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66002025-04-30T06:46:16.623ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
66012025-04-30T06:46:16.624ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66022025-04-30T06:46:16.636ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66032025-04-30T06:46:16.636ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66042025-04-30T06:46:16.640ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66052025-04-30T06:46:16.640ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66062025-04-30T06:46:16.640ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66072025-04-30T06:46:16.640ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66082025-04-30T06:46:16.672ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
66092025-04-30T06:46:16.680ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66102025-04-30T06:46:16.680ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66112025-04-30T06:46:16.680ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 384.691483ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66122025-04-30T06:46:16.680ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66132025-04-30T06:46:16.680ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66142025-04-30T06:46:16.680ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66152025-04-30T06:46:16.680ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66162025-04-30T06:46:16.680ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66172025-04-30T06:46:16.687ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66182025-04-30T06:46:16.692ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66192025-04-30T06:46:16.700ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66202025-04-30T06:46:16.700ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66212025-04-30T06:46:16.700ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66222025-04-30T06:46:16.701ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66232025-04-30T06:46:16.707ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
66242025-04-30T06:46:16.707ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66252025-04-30T06:46:16.708ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d8 f3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49458 req_id = 3fae4879-2487-41c5-8672-e330847a0f0e
66262025-04-30T06:46:16.708ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55539,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49458 req_id = 3fae4879-2487-41c5-8672-e330847a0f0e
66272025-04-30T06:46:16.708ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
66282025-04-30T06:46:16.709ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66292025-04-30T06:46:16.709ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49458 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3fae4879-2487-41c5-8672-e330847a0f0e
66302025-04-30T06:46:16.709ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4441, tv_nsec: 508428762 } })
66312025-04-30T06:46:16.709ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
66322025-04-30T06:46:16.709ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0e 0e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51666 req_id = 45063cff-1b67-430b-a963-f3767c520c8a
66332025-04-30T06:46:16.710ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3598,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51666 req_id = 45063cff-1b67-430b-a963-f3767c520c8a
66342025-04-30T06:46:16.710ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66352025-04-30T06:46:16.710ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51666 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 45063cff-1b67-430b-a963-f3767c520c8a
66362025-04-30T06:46:16.770ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66372025-04-30T06:46:16.770ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66382025-04-30T06:46:16.770ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66392025-04-30T06:46:16.770ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66402025-04-30T06:46:16.770ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
66412025-04-30T06:46:16.771ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66422025-04-30T06:46:16.771ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2f 96 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c /............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54516 req_id = 73013a02-20ad-4ec1-b036-e9fd6c9d0c1b
66432025-04-30T06:46:16.772ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54516 req_id = 73013a02-20ad-4ec1-b036-e9fd6c9d0c1b
66442025-04-30T06:46:16.772ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
66452025-04-30T06:46:16.773ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12182, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54516 req_id = 73013a02-20ad-4ec1-b036-e9fd6c9d0c1b
66462025-04-30T06:46:16.775ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
66472025-04-30T06:46:16.775ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
66482025-04-30T06:46:16.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
66492025-04-30T06:46:16.776ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66502025-04-30T06:46:16.782ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66512025-04-30T06:46:16.782ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66522025-04-30T06:46:16.782ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
66532025-04-30T06:46:16.782ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66542025-04-30T06:46:16.784ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3bf1db-62b0-4b23-82b9-0a5e6484c137", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66552025-04-30T06:46:16.784ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
66562025-04-30T06:46:16.786ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66572025-04-30T06:46:16.786ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
66582025-04-30T06:46:16.787ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66592025-04-30T06:46:16.789ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dec8755f-1ff7-41fd-9a8d-0983b7f2393f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66602025-04-30T06:46:16.789ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
66612025-04-30T06:46:16.794ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "099e3232-2c91-48bb-ab92-e24aead9a4a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66622025-04-30T06:46:16.794ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
66632025-04-30T06:46:16.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "210ed84e-a5da-42af-a25a-e2b62c9cdd7a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66642025-04-30T06:46:16.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
66652025-04-30T06:46:16.805ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b660bd-fb75-4b32-885f-6a7c4a71b58f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66662025-04-30T06:46:16.805ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
66672025-04-30T06:46:16.808ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c87f5a2e-eda0-4930-ae24-9e57aa8cf04f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66682025-04-30T06:46:16.808ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
66692025-04-30T06:46:16.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe91034-14b9-4c4a-8319-cfd3f0da2dac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66702025-04-30T06:46:16.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
66712025-04-30T06:46:16.813ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7035c442-111a-4d14-b092-a5a42233f978", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66722025-04-30T06:46:16.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
66732025-04-30T06:46:16.815ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fc2e588-a73a-443b-ba26-75d7981f685c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66742025-04-30T06:46:16.815ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
66752025-04-30T06:46:16.817ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b359a577-9024-46c3-8a9c-4f48b2083fc8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66762025-04-30T06:46:16.818ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
66772025-04-30T06:46:16.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcdf9602-fc2d-433a-8d21-1fc771d57dff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66782025-04-30T06:46:16.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
66792025-04-30T06:46:16.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48ff30e-b811-4755-8c7a-07b405b0631e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66802025-04-30T06:46:16.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
66812025-04-30T06:46:16.822ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "283a48c0-2482-4fdb-9f2c-e02d7ea79ddf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66822025-04-30T06:46:16.822ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
66832025-04-30T06:46:16.834ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
66842025-04-30T06:46:16.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbda7608-2d3f-4d98-8d82-37d31644d9cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66852025-04-30T06:46:16.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
66862025-04-30T06:46:16.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea078a6a-ad64-4e41-8fca-3c0487b87220", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66872025-04-30T06:46:16.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
66882025-04-30T06:46:16.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c551fa87-4436-4bb5-a7b9-a04191feed9f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66892025-04-30T06:46:16.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
66902025-04-30T06:46:16.846ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48966f5a-3ec4-42ad-9d2f-3ed0895589aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66912025-04-30T06:46:16.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
66922025-04-30T06:46:16.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5bf247f-f1d4-4c03-8745-7ebb887bca01", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66932025-04-30T06:46:16.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
66942025-04-30T06:46:16.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54f44f66-67df-44ca-8158-bc055ff67f08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66952025-04-30T06:46:16.847ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
66962025-04-30T06:46:16.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e86e8f-ecc0-4983-8f33-e81db95f7137", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66972025-04-30T06:46:16.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
66982025-04-30T06:46:16.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23fba548-aff8-43ec-b381-2f87e4bd0a3a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
66992025-04-30T06:46:16.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
67002025-04-30T06:46:16.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99baf930-e243-44af-968c-3fd8b2ef9659", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67012025-04-30T06:46:16.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
67022025-04-30T06:46:16.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c126875d-8eae-455b-b794-7031d01a30c4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67032025-04-30T06:46:16.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
67042025-04-30T06:46:16.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0094bd18-36c9-4e10-9b54-fd3a79e9beb9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67052025-04-30T06:46:16.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
67062025-04-30T06:46:16.856ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da96ec2b-1c37-42dc-b8a0-e9d4b3b487af", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67072025-04-30T06:46:16.856ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
67082025-04-30T06:46:16.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68f7b2db-c2f7-45f5-bb39-d8ebf680a7ef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67092025-04-30T06:46:16.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
67102025-04-30T06:46:16.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "449d78ec-8728-4fee-8246-c6f3dff9dd57", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67112025-04-30T06:46:16.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
67122025-04-30T06:46:16.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bd7caba-a96a-4a64-9565-8d955d44ab59", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67132025-04-30T06:46:16.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
67142025-04-30T06:46:16.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5903f4d-05b3-4d0a-a27d-e97e0cb6bea8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67152025-04-30T06:46:16.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
67162025-04-30T06:46:16.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b75fc69-c58a-4794-b098-ddfaea4ecffc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67172025-04-30T06:46:16.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
67182025-04-30T06:46:16.868ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da659f58-c19b-42b7-ad3a-0118936dba43", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67192025-04-30T06:46:16.868ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
67202025-04-30T06:46:16.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45025850-9009-4711-9c1e-2956c10096e7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67212025-04-30T06:46:16.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
67222025-04-30T06:46:16.871ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0ccba71-256f-41dd-840d-ce7f50267a47", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67232025-04-30T06:46:16.871ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
67242025-04-30T06:46:16.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dcb307c-880d-4050-9885-b73b7dee3a49", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67252025-04-30T06:46:16.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
67262025-04-30T06:46:16.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9fbfa30-0724-4bb9-9be5-b25cc792bc80", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67272025-04-30T06:46:16.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
67282025-04-30T06:46:16.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3df1c5ba-7c26-41ee-8479-dba05a400287", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67292025-04-30T06:46:16.876ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
67302025-04-30T06:46:16.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6f6c18b-1ca7-4cdc-ac58-2d41d5aa25db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67312025-04-30T06:46:16.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
67322025-04-30T06:46:16.885ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67332025-04-30T06:46:16.885ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67342025-04-30T06:46:16.885ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67352025-04-30T06:46:16.885ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
67362025-04-30T06:46:16.886ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67372025-04-30T06:46:16.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfedc5cf-e8e3-4fb0-9907-1e428edfcb1e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67382025-04-30T06:46:16.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
67392025-04-30T06:46:16.889ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67402025-04-30T06:46:16.897ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67412025-04-30T06:46:16.897ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67422025-04-30T06:46:16.897ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
67432025-04-30T06:46:16.898ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67442025-04-30T06:46:16.902ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67452025-04-30T06:46:16.902ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
67462025-04-30T06:46:16.902ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67472025-04-30T06:46:16.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ff1f792-d656-437b-8793-e007f2dab637", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67482025-04-30T06:46:16.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
67492025-04-30T06:46:16.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb5e044-0bd6-46d2-9ebf-ed4404328d3e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67502025-04-30T06:46:16.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
67512025-04-30T06:46:16.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "737bb074-7ef1-45b7-b562-cd8ed7649380", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67522025-04-30T06:46:16.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
67532025-04-30T06:46:16.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1538e45-363e-4b17-a376-a7c017ca1a09", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67542025-04-30T06:46:16.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
67552025-04-30T06:46:16.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f305d965-0642-48f3-b462-d028b31d90cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67562025-04-30T06:46:16.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
67572025-04-30T06:46:16.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "613181aa-626c-4468-8be3-1713a136f418", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67582025-04-30T06:46:16.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
67592025-04-30T06:46:16.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a87bdf82-b546-4e62-8835-bd0ed91a92f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67602025-04-30T06:46:16.922ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67612025-04-30T06:46:16.922ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67622025-04-30T06:46:16.922ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67632025-04-30T06:46:16.922ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67642025-04-30T06:46:16.922ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67652025-04-30T06:46:16.933ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67662025-04-30T06:46:16.937ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67672025-04-30T06:46:16.938ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67682025-04-30T06:46:16.938ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67692025-04-30T06:46:16.938ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67702025-04-30T06:46:16.938ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67712025-04-30T06:46:16.938ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67722025-04-30T06:46:16.942ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67732025-04-30T06:46:16.955ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67742025-04-30T06:46:16.955ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67752025-04-30T06:46:16.955ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67762025-04-30T06:46:16.955ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67772025-04-30T06:46:17.019ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
67782025-04-30T06:46:17.020ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
67792025-04-30T06:46:17.022ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:48141 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
67802025-04-30T06:46:17.022ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67812025-04-30T06:46:17.023ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67822025-04-30T06:46:17.023ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
67832025-04-30T06:46:17.024ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
67842025-04-30T06:46:17.026ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
67852025-04-30T06:46:17.028ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:48141 req_id = 14a48195-88eb-4e23-b4fe-d88acbcd01f8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67862025-04-30T06:46:17.029ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 891 local_addr = [::1]:54590 method = GET remote_addr = [::1]:48141 req_id = 14a48195-88eb-4e23-b4fe-d88acbcd01f8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67872025-04-30T06:46:17.031ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
67882025-04-30T06:46:17.031ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
67892025-04-30T06:46:17.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "14a48195-88eb-4e23-b4fe-d88acbcd01f8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
67902025-04-30T06:46:17.034ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67912025-04-30T06:46:17.034ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67922025-04-30T06:46:17.034ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
67932025-04-30T06:46:17.036ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67942025-04-30T06:46:17.036ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67952025-04-30T06:46:17.036ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67962025-04-30T06:46:17.036ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67972025-04-30T06:46:17.037ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67982025-04-30T06:46:17.040ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
67992025-04-30T06:46:17.041ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68002025-04-30T06:46:17.041ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68012025-04-30T06:46:17.045ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68022025-04-30T06:46:17.045ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68032025-04-30T06:46:17.046ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68042025-04-30T06:46:17.046ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68052025-04-30T06:46:17.081ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68062025-04-30T06:46:17.081ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68072025-04-30T06:46:17.082ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68082025-04-30T06:46:17.082ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
68092025-04-30T06:46:17.083ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68102025-04-30T06:46:17.086ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68112025-04-30T06:46:17.103ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68122025-04-30T06:46:17.115ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68132025-04-30T06:46:17.115ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
68142025-04-30T06:46:17.125ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68152025-04-30T06:46:17.136ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
68162025-04-30T06:46:17.153ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68172025-04-30T06:46:17.153ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68182025-04-30T06:46:17.157ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 476.842026ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68192025-04-30T06:46:17.157ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68202025-04-30T06:46:17.157ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68212025-04-30T06:46:17.157ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68222025-04-30T06:46:17.157ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68232025-04-30T06:46:17.158ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68242025-04-30T06:46:17.159ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68252025-04-30T06:46:17.168ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68262025-04-30T06:46:17.168ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
68272025-04-30T06:46:17.169ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68282025-04-30T06:46:17.169ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68292025-04-30T06:46:17.178ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68302025-04-30T06:46:17.178ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68312025-04-30T06:46:17.178ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68322025-04-30T06:46:17.179ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68332025-04-30T06:46:17.182ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
68342025-04-30T06:46:17.182ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68352025-04-30T06:46:17.184ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e0 a7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59458 req_id = 91ca33e2-edc7-4b31-9a29-b23321361592
68362025-04-30T06:46:17.184ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59458 req_id = 91ca33e2-edc7-4b31-9a29-b23321361592
68372025-04-30T06:46:17.184ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
68382025-04-30T06:46:17.185ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68392025-04-30T06:46:17.185ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59458 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 91ca33e2-edc7-4b31-9a29-b23321361592
68402025-04-30T06:46:17.186ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4441, tv_nsec: 985101290 } })
68412025-04-30T06:46:17.186ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
68422025-04-30T06:46:17.186ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 04 79 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49848 req_id = 28288d60-ceee-46cc-aa46-c73b5f865a1e
68432025-04-30T06:46:17.186ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1145,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49848 req_id = 28288d60-ceee-46cc-aa46-c73b5f865a1e
68442025-04-30T06:46:17.187ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68452025-04-30T06:46:17.187ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49848 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 28288d60-ceee-46cc-aa46-c73b5f865a1e
68462025-04-30T06:46:17.319ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68472025-04-30T06:46:17.320ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 48 29 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c H)..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61299 req_id = 46e31f8e-62df-44ac-b020-0e01c3aa75a1
68482025-04-30T06:46:17.320ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61299 req_id = 46e31f8e-62df-44ac-b020-0e01c3aa75a1
68492025-04-30T06:46:17.321ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
68502025-04-30T06:46:17.322ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18473, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61299 req_id = 46e31f8e-62df-44ac-b020-0e01c3aa75a1
68512025-04-30T06:46:17.323ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
68522025-04-30T06:46:17.323ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
68532025-04-30T06:46:17.323ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
68542025-04-30T06:46:17.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa982551-3c6b-4244-9f79-5402a5be81fd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68552025-04-30T06:46:17.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
68562025-04-30T06:46:17.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c93d2da-e239-4985-aae0-507153b61596", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68572025-04-30T06:46:17.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
68582025-04-30T06:46:17.341ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9502415d-8cfb-4fcf-ad17-3db094af5806", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68592025-04-30T06:46:17.342ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
68602025-04-30T06:46:17.346ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d9c6c5-9c07-4c1b-a6e5-2f2686faac2f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68612025-04-30T06:46:17.346ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
68622025-04-30T06:46:17.349ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
68632025-04-30T06:46:17.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1587d1d8-602a-4de6-b22f-7691a2f5549e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68642025-04-30T06:46:17.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
68652025-04-30T06:46:17.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ace5247-3cfe-47a5-8a26-a270fda03064", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68662025-04-30T06:46:17.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
68672025-04-30T06:46:17.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37f58146-cf1a-46bf-b61b-a794356e7c44", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68682025-04-30T06:46:17.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
68692025-04-30T06:46:17.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "466b79f0-9670-4f12-a24c-c11c3b6ac300", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68702025-04-30T06:46:17.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
68712025-04-30T06:46:17.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b77d9f59-3181-4299-aaa5-79e19534872b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68722025-04-30T06:46:17.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
68732025-04-30T06:46:17.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3631581a-288c-4e46-8aab-a108a6bf556f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68742025-04-30T06:46:17.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
68752025-04-30T06:46:17.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e5d584-2cc1-49ae-bbc8-3416f082c0af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68762025-04-30T06:46:17.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
68772025-04-30T06:46:17.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41c25c02-22b7-4bb6-898d-511595747480", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68782025-04-30T06:46:17.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
68792025-04-30T06:46:17.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5021b9a5-7b63-4eb1-8eb3-9c562c24cfe3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68802025-04-30T06:46:17.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
68812025-04-30T06:46:17.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca6db548-7d81-4f92-abad-a0ae7d0e0f00", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68822025-04-30T06:46:17.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
68832025-04-30T06:46:17.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67f521e-a0e4-463b-b346-c0737093d28c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68842025-04-30T06:46:17.389ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
68852025-04-30T06:46:17.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4ecbedf-f9ce-44fc-ad06-655615c17501", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68862025-04-30T06:46:17.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
68872025-04-30T06:46:17.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "949cdc4b-369b-4bf8-82d2-f4768c96a00d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68882025-04-30T06:46:17.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
68892025-04-30T06:46:17.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8494848f-5be6-4adf-98b1-ca38f2dbb1ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68902025-04-30T06:46:17.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
68912025-04-30T06:46:17.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e0b6e89-a670-4dff-a96d-90c7449f8528", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68922025-04-30T06:46:17.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
68932025-04-30T06:46:17.398ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58e7ffef-1c80-429d-a323-19d39b8eabd4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68942025-04-30T06:46:17.398ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
68952025-04-30T06:46:17.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9812cb5f-127f-493f-8ea0-f847fd27a3db", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
68962025-04-30T06:46:17.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
68972025-04-30T06:46:17.400ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68982025-04-30T06:46:17.400ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68992025-04-30T06:46:17.400ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69002025-04-30T06:46:17.400ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69012025-04-30T06:46:17.401ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69022025-04-30T06:46:17.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ecd118e-1d9d-4683-8e29-74ab1ec2f08b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69032025-04-30T06:46:17.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
69042025-04-30T06:46:17.403ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c4f92ce-27b5-42c8-9566-e722e746881a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69052025-04-30T06:46:17.403ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
69062025-04-30T06:46:17.404ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69072025-04-30T06:46:17.414ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69082025-04-30T06:46:17.415ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69092025-04-30T06:46:17.415ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69102025-04-30T06:46:17.416ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69112025-04-30T06:46:17.419ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6450667-d2e4-4b10-b370-eb19790492a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69122025-04-30T06:46:17.419ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
69132025-04-30T06:46:17.420ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69142025-04-30T06:46:17.420ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69152025-04-30T06:46:17.421ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69162025-04-30T06:46:17.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a9d412d-7529-42eb-a395-8d68ad2ccc2b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69172025-04-30T06:46:17.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
69182025-04-30T06:46:17.461ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ca8e43-f62a-40cd-bdc0-b5d8157b90fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69192025-04-30T06:46:17.461ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
69202025-04-30T06:46:17.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e1a8814-ed0b-4d91-847f-97173d7419e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69212025-04-30T06:46:17.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
69222025-04-30T06:46:17.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b01c486-c104-44b3-96cb-341ae5fbf9d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69232025-04-30T06:46:17.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
69242025-04-30T06:46:17.477ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5586b47-bc33-41fa-9678-07699d869030", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69252025-04-30T06:46:17.477ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
69262025-04-30T06:46:17.480ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0e52f9-2684-4a29-8cdb-98020afb9faf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69272025-04-30T06:46:17.480ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
69282025-04-30T06:46:17.481ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
69292025-04-30T06:46:17.483ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a06e8ac3-ac32-45c6-8548-d6a40ac2f0b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69302025-04-30T06:46:17.483ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
69312025-04-30T06:46:17.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da0fb145-3dde-4ad5-86ee-22aa47d5f52e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69322025-04-30T06:46:17.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
69332025-04-30T06:46:17.485ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24e749a0-8e78-433b-b34f-f3d268034dd8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69342025-04-30T06:46:17.485ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
69352025-04-30T06:46:17.486ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "613a590d-3d3e-409f-9b91-44647279e56e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69362025-04-30T06:46:17.486ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
69372025-04-30T06:46:17.487ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ce42299-3c60-40c4-a37d-c55fd2872238", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69382025-04-30T06:46:17.487ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
69392025-04-30T06:46:17.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80f04e6a-3c80-41bc-9f30-685985854dbb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69402025-04-30T06:46:17.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
69412025-04-30T06:46:17.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db297bfa-dda7-496c-b3c7-1c953f941f45", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69422025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
69432025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c97d082a-d431-4f13-8744-bbe73d3b843e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69442025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
69452025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bc4d6af-1b2b-4238-98eb-ac00d4d68c8a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69462025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
69472025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ed0acc0-55ac-42a6-9c74-0bf06a1acf41", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69482025-04-30T06:46:17.507ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
69492025-04-30T06:46:17.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eeecfb8-bd3c-481c-9684-c30ecf737e1d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69502025-04-30T06:46:17.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
69512025-04-30T06:46:17.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74e386a-0568-483d-a274-819d4a1be997", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69522025-04-30T06:46:17.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
69532025-04-30T06:46:17.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b466387e-d5cd-4cb0-ae99-204e1bbc16f5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69542025-04-30T06:46:17.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
69552025-04-30T06:46:17.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f7bb0af-ed14-4b27-a1a4-e9883e02bee7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69562025-04-30T06:46:17.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
69572025-04-30T06:46:17.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c58c94d-1a32-4c52-ac4e-93f4c93fdd10", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
69582025-04-30T06:46:17.514ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69592025-04-30T06:46:17.514ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69602025-04-30T06:46:17.514ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69612025-04-30T06:46:17.514ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69622025-04-30T06:46:17.514ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69632025-04-30T06:46:17.524ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69642025-04-30T06:46:17.531ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69652025-04-30T06:46:17.531ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69662025-04-30T06:46:17.531ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69672025-04-30T06:46:17.531ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69682025-04-30T06:46:17.531ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69692025-04-30T06:46:17.531ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69702025-04-30T06:46:17.532ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69712025-04-30T06:46:17.532ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69722025-04-30T06:46:17.532ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69732025-04-30T06:46:17.532ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69742025-04-30T06:46:17.533ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69752025-04-30T06:46:17.534ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69762025-04-30T06:46:17.539ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69772025-04-30T06:46:17.539ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69782025-04-30T06:46:17.539ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69792025-04-30T06:46:17.540ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69802025-04-30T06:46:17.543ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69812025-04-30T06:46:17.559ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69822025-04-30T06:46:17.559ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69832025-04-30T06:46:17.560ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69842025-04-30T06:46:17.560ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69852025-04-30T06:46:17.561ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
69862025-04-30T06:46:17.562ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69872025-04-30T06:46:17.563ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9f89c6ab-55be-4296-97d4-1f6ef52882c9 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69882025-04-30T06:46:17.565ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9f89c6ab-55be-4296-97d4-1f6ef52882c9 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69892025-04-30T06:46:17.565ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9f89c6ab-55be-4296-97d4-1f6ef52882c9 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69902025-04-30T06:46:17.566ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9f89c6ab-55be-4296-97d4-1f6ef52882c9 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69912025-04-30T06:46:17.645ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
69922025-04-30T06:46:17.645ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
69932025-04-30T06:46:17.649ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 85823 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9f89c6ab-55be-4296-97d4-1f6ef52882c9 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
69942025-04-30T06:46:17.651ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69952025-04-30T06:46:17.651ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
69962025-04-30T06:46:17.652ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69972025-04-30T06:46:17.652ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:65281 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
69982025-04-30T06:46:17.653ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69992025-04-30T06:46:17.653ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f89c6ab-55be-4296-97d4-1f6ef52882c9", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
70002025-04-30T06:46:17.653ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70012025-04-30T06:46:17.654ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70022025-04-30T06:46:17.657ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = cd87d4db-fbd5-4841-a1ad-f78c428c333d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70032025-04-30T06:46:17.657ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:65281 req_id = 95524e9f-58dc-418d-a305-879208d477b3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70042025-04-30T06:46:17.665ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = cd87d4db-fbd5-4841-a1ad-f78c428c333d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70052025-04-30T06:46:17.665ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = cd87d4db-fbd5-4841-a1ad-f78c428c333d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70062025-04-30T06:46:17.666ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = cd87d4db-fbd5-4841-a1ad-f78c428c333d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70072025-04-30T06:46:17.722ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 64276 local_addr = [::1]:54590 method = GET remote_addr = [::1]:65281 req_id = 95524e9f-58dc-418d-a305-879208d477b3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70082025-04-30T06:46:17.723ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70092025-04-30T06:46:17.724ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70102025-04-30T06:46:17.724ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70112025-04-30T06:46:17.724ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70122025-04-30T06:46:17.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "95524e9f-58dc-418d-a305-879208d477b3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
70132025-04-30T06:46:17.727ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70142025-04-30T06:46:17.727ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70152025-04-30T06:46:17.727ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
70162025-04-30T06:46:17.729ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70172025-04-30T06:46:17.729ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70182025-04-30T06:46:17.730ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70192025-04-30T06:46:17.730ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70202025-04-30T06:46:17.731ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70212025-04-30T06:46:17.735ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 77752 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = cd87d4db-fbd5-4841-a1ad-f78c428c333d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70222025-04-30T06:46:17.736ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd87d4db-fbd5-4841-a1ad-f78c428c333d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:16 GMT"} })
70232025-04-30T06:46:17.736ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
70242025-04-30T06:46:17.736ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70252025-04-30T06:46:17.736ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
70262025-04-30T06:46:17.736ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70272025-04-30T06:46:17.742ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70282025-04-30T06:46:17.743ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70292025-04-30T06:46:17.743ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70302025-04-30T06:46:17.744ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70312025-04-30T06:46:17.811ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
70322025-04-30T06:46:17.816ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
70332025-04-30T06:46:17.817ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
70342025-04-30T06:46:17.817ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 659.81096ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
70352025-04-30T06:46:17.817ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
70362025-04-30T06:46:17.817ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70372025-04-30T06:46:17.817ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70382025-04-30T06:46:17.817ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70392025-04-30T06:46:17.817ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70402025-04-30T06:46:17.818ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70412025-04-30T06:46:17.823ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70422025-04-30T06:46:17.824ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70432025-04-30T06:46:17.824ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70442025-04-30T06:46:17.824ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70452025-04-30T06:46:17.825ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70462025-04-30T06:46:17.829ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
70472025-04-30T06:46:17.829ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70482025-04-30T06:46:17.829ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bc 19 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55911 req_id = 5e1ec960-57a2-49a9-a8da-2a00a7289a2e
70492025-04-30T06:46:17.830ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48153,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55911 req_id = 5e1ec960-57a2-49a9-a8da-2a00a7289a2e
70502025-04-30T06:46:17.830ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
70512025-04-30T06:46:17.830ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70522025-04-30T06:46:17.830ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55911 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5e1ec960-57a2-49a9-a8da-2a00a7289a2e
70532025-04-30T06:46:17.830ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4442, tv_nsec: 629841533 } })
70542025-04-30T06:46:17.831ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
70552025-04-30T06:46:17.831ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 43 ab 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e C...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49833 req_id = fdf712f6-8492-4406-83b4-8e835e849ae1
70562025-04-30T06:46:17.831ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17323,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49833 req_id = fdf712f6-8492-4406-83b4-8e835e849ae1
70572025-04-30T06:46:17.831ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70582025-04-30T06:46:17.831ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49833 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fdf712f6-8492-4406-83b4-8e835e849ae1
70592025-04-30T06:46:17.905ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70602025-04-30T06:46:17.905ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70612025-04-30T06:46:17.905ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70622025-04-30T06:46:17.905ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70632025-04-30T06:46:17.905ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
70642025-04-30T06:46:17.906ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70652025-04-30T06:46:17.906ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 35 ba 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 5............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51482 req_id = e037f428-9e61-4472-ac6a-5962e503c961
70662025-04-30T06:46:17.907ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13754,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51482 req_id = e037f428-9e61-4472-ac6a-5962e503c961
70672025-04-30T06:46:17.907ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
70682025-04-30T06:46:17.907ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13754, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51482 req_id = e037f428-9e61-4472-ac6a-5962e503c961
70692025-04-30T06:46:17.907ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
70702025-04-30T06:46:17.907ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
70712025-04-30T06:46:17.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
70722025-04-30T06:46:17.911ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70732025-04-30T06:46:17.918ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70742025-04-30T06:46:17.918ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70752025-04-30T06:46:17.918ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
70762025-04-30T06:46:17.921ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70772025-04-30T06:46:17.933ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70782025-04-30T06:46:17.933ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
70792025-04-30T06:46:17.934ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70802025-04-30T06:46:17.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c90844d-6838-4731-9aba-dc438c1ff131", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70812025-04-30T06:46:17.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
70822025-04-30T06:46:17.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac49d280-3f7f-4056-80e8-77aa0cfd03f9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70832025-04-30T06:46:17.941ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
70842025-04-30T06:46:17.945ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f93e16ff-bea2-4015-9a3d-9eed15f1c2ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70852025-04-30T06:46:17.945ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
70862025-04-30T06:46:17.950ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "630e3959-273f-4a18-9a66-a7b602e6e381", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70872025-04-30T06:46:17.950ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
70882025-04-30T06:46:17.954ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "858b1d35-ff85-42f8-b2d8-f7b541fafddd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70892025-04-30T06:46:17.954ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
70902025-04-30T06:46:17.967ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24f77828-0409-471c-a2bb-ef3ad9e51136", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70912025-04-30T06:46:17.967ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
70922025-04-30T06:46:17.973ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df75d101-c7d1-4954-bf04-73f9a83efb2c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
70932025-04-30T06:46:17.974ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
70942025-04-30T06:46:17.988ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70952025-04-30T06:46:17.988ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
70962025-04-30T06:46:17.988ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
70972025-04-30T06:46:17.991ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
70982025-04-30T06:46:17.992ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
70992025-04-30T06:46:17.996ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d02122-8841-43e6-91a7-08ad73f750a3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71002025-04-30T06:46:17.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
71012025-04-30T06:46:17.997ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
71022025-04-30T06:46:17.997ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71032025-04-30T06:46:18.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abcd47a1-6158-4ed8-b236-c9f1766496bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71042025-04-30T06:46:18.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
71052025-04-30T06:46:18.004ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
71062025-04-30T06:46:18.004ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71072025-04-30T06:46:18.004ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "906c6c35-3b2b-481c-900e-9d60db6b8866", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71082025-04-30T06:46:18.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
71092025-04-30T06:46:18.015ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0639c7a-cebe-4c61-bedc-2c4f0e60f67c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71102025-04-30T06:46:18.015ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
71112025-04-30T06:46:18.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f6c3f8-b16d-4e6a-b716-92b80152ff4a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71122025-04-30T06:46:18.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
71132025-04-30T06:46:18.025ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fd685ab-7cda-44a4-8fa1-165bd72ef53f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71142025-04-30T06:46:18.026ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
71152025-04-30T06:46:18.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd31426-afba-4df1-b9e6-4297f74c640b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71162025-04-30T06:46:18.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
71172025-04-30T06:46:18.031ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a232fc02-03c7-49db-9dc5-0fbe8584c620", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71182025-04-30T06:46:18.031ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
71192025-04-30T06:46:18.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f59a11de-021c-4ba9-93dd-3c24931582d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71202025-04-30T06:46:18.036ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
71212025-04-30T06:46:18.041ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2575721-621e-4cfe-87d6-a1b5df071fd5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71222025-04-30T06:46:18.050ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
71232025-04-30T06:46:18.051ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71242025-04-30T06:46:18.052ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71252025-04-30T06:46:18.052ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71262025-04-30T06:46:18.052ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71272025-04-30T06:46:18.053ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71282025-04-30T06:46:18.058ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71292025-04-30T06:46:18.064ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71302025-04-30T06:46:18.064ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71312025-04-30T06:46:18.064ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71322025-04-30T06:46:18.065ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71332025-04-30T06:46:18.065ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2e512ac-f2ae-4bbb-b092-1d1990f6903b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71342025-04-30T06:46:18.065ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
71352025-04-30T06:46:18.068ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71362025-04-30T06:46:18.068ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71372025-04-30T06:46:18.069ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71382025-04-30T06:46:18.070ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "adbf6ff2-3c3d-4b60-b429-c16592df8492", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71392025-04-30T06:46:18.070ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
71402025-04-30T06:46:18.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82cf9b2c-4c01-41ef-8428-35e68d9f0814", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71412025-04-30T06:46:18.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
71422025-04-30T06:46:18.072ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "680da0cf-b927-4985-8b8e-ad90b5967387", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71432025-04-30T06:46:18.072ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
71442025-04-30T06:46:18.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc776ae-cc55-4997-965e-b89057316fc2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71452025-04-30T06:46:18.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
71462025-04-30T06:46:18.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64727221-a524-42a3-a356-5da3895e5f5f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71472025-04-30T06:46:18.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
71482025-04-30T06:46:18.077ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ac1d0d3-9674-4583-ae7f-237a8150ad37", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71492025-04-30T06:46:18.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
71502025-04-30T06:46:18.082ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e2becbe-5c50-4d30-a38a-a247ddeffc19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71512025-04-30T06:46:18.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
71522025-04-30T06:46:18.131ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
71532025-04-30T06:46:18.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "673beade-3075-49a4-931d-4d6b4f71177f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71542025-04-30T06:46:18.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
71552025-04-30T06:46:18.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ec8b721-9a8f-4f6d-801b-7002ddd1c039", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71562025-04-30T06:46:18.164ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
71572025-04-30T06:46:18.196ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
71582025-04-30T06:46:18.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc472c92-3742-4487-852a-85748fc052cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71592025-04-30T06:46:18.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
71602025-04-30T06:46:18.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a126ce-d7eb-4794-8502-656ada8f6973", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71612025-04-30T06:46:18.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
71622025-04-30T06:46:18.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26094b9-98e5-43af-b9f0-12e6e4788515", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71632025-04-30T06:46:18.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
71642025-04-30T06:46:18.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "159cab3c-a9fd-4bf5-8cfd-e1473e6a6155", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71652025-04-30T06:46:18.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
71662025-04-30T06:46:18.213ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c76a26f9-55dc-410d-b3ab-8791ce362457", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71672025-04-30T06:46:18.213ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
71682025-04-30T06:46:18.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b83c5897-07ed-4cb0-8b08-e80337fa6cfd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71692025-04-30T06:46:18.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
71702025-04-30T06:46:18.217ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8be02b1-7888-4a65-9b35-98c66a9e22ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71712025-04-30T06:46:18.217ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
71722025-04-30T06:46:18.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d71672d4-944e-4da1-a03f-dc659f996618", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71732025-04-30T06:46:18.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
71742025-04-30T06:46:18.229ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7ce03fe-8830-443d-9f65-98f4dec0f423", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71752025-04-30T06:46:18.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
71762025-04-30T06:46:18.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "678253b7-b892-4e17-85c5-732cdb25032c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71772025-04-30T06:46:18.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
71782025-04-30T06:46:18.249ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71792025-04-30T06:46:18.249ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71802025-04-30T06:46:18.249ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71812025-04-30T06:46:18.249ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71822025-04-30T06:46:18.251ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71832025-04-30T06:46:18.262ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71842025-04-30T06:46:18.268ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71852025-04-30T06:46:18.268ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71862025-04-30T06:46:18.268ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71872025-04-30T06:46:18.268ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71882025-04-30T06:46:18.276ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d07b785-16b0-4005-8b38-977754e9c8ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71892025-04-30T06:46:18.276ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
71902025-04-30T06:46:18.276ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71912025-04-30T06:46:18.276ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
71922025-04-30T06:46:18.277ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71932025-04-30T06:46:18.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0711e16a-e8a3-405b-b506-1a55fa44740d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71942025-04-30T06:46:18.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
71952025-04-30T06:46:18.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "936b691c-ed6b-4a76-bacf-43cbfbc8a1b3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71962025-04-30T06:46:18.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
71972025-04-30T06:46:18.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6477e0f-a780-4064-bf22-b7af4c2da340", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
71982025-04-30T06:46:18.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
71992025-04-30T06:46:18.286ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c7f5ac-6e48-4d7e-907b-3b4e2f4ab868", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72002025-04-30T06:46:18.286ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
72012025-04-30T06:46:18.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70f175b4-a10d-405e-9a64-a45610583499", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72022025-04-30T06:46:18.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
72032025-04-30T06:46:18.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb8c150-d86d-4c2c-9f17-366655548fca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72042025-04-30T06:46:18.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
72052025-04-30T06:46:18.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a46a14b-3095-41de-88c2-4ac3b6882738", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72062025-04-30T06:46:18.293ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72072025-04-30T06:46:18.293ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72082025-04-30T06:46:18.293ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72092025-04-30T06:46:18.293ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72102025-04-30T06:46:18.293ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72112025-04-30T06:46:18.296ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72122025-04-30T06:46:18.300ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72132025-04-30T06:46:18.301ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72142025-04-30T06:46:18.301ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72152025-04-30T06:46:18.301ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72162025-04-30T06:46:18.301ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72172025-04-30T06:46:18.301ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72182025-04-30T06:46:18.306ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72192025-04-30T06:46:18.310ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72202025-04-30T06:46:18.310ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72212025-04-30T06:46:18.310ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72222025-04-30T06:46:18.311ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72232025-04-30T06:46:18.375ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
72242025-04-30T06:46:18.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
72252025-04-30T06:46:18.375ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:61412 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72262025-04-30T06:46:18.376ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:61412 req_id = 3d72560e-7c94-4be2-8023-8bef1ad53197 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72272025-04-30T06:46:18.384ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 8301 local_addr = [::1]:54590 method = GET remote_addr = [::1]:61412 req_id = 3d72560e-7c94-4be2-8023-8bef1ad53197 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72282025-04-30T06:46:18.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d72560e-7c94-4be2-8023-8bef1ad53197", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72292025-04-30T06:46:18.395ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72302025-04-30T06:46:18.439ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72312025-04-30T06:46:18.440ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
72322025-04-30T06:46:18.443ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72332025-04-30T06:46:18.443ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72342025-04-30T06:46:18.443ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72352025-04-30T06:46:18.443ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72362025-04-30T06:46:18.444ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72372025-04-30T06:46:18.445ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
72382025-04-30T06:46:18.465ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72392025-04-30T06:46:18.469ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72402025-04-30T06:46:18.470ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72412025-04-30T06:46:18.470ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72422025-04-30T06:46:18.470ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72432025-04-30T06:46:18.495ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72442025-04-30T06:46:18.495ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72452025-04-30T06:46:18.495ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72462025-04-30T06:46:18.495ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
72472025-04-30T06:46:18.496ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72482025-04-30T06:46:18.501ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72492025-04-30T06:46:18.507ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72502025-04-30T06:46:18.507ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72512025-04-30T06:46:18.507ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
72522025-04-30T06:46:18.507ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72532025-04-30T06:46:18.549ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72542025-04-30T06:46:18.549ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
72552025-04-30T06:46:18.550ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72562025-04-30T06:46:18.551ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
72572025-04-30T06:46:18.551ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
72582025-04-30T06:46:18.551ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 734.318332ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
72592025-04-30T06:46:18.551ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
72602025-04-30T06:46:18.552ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72612025-04-30T06:46:18.552ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72622025-04-30T06:46:18.552ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72632025-04-30T06:46:18.552ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72642025-04-30T06:46:18.553ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72652025-04-30T06:46:18.555ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72662025-04-30T06:46:18.575ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72672025-04-30T06:46:18.575ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72682025-04-30T06:46:18.575ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72692025-04-30T06:46:18.575ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72702025-04-30T06:46:18.583ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
72712025-04-30T06:46:18.585ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
72722025-04-30T06:46:18.588ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d2 bb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55018 req_id = f6a862d0-5bc6-4f75-bd19-e844499743c0
72732025-04-30T06:46:18.589ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53947,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55018 req_id = f6a862d0-5bc6-4f75-bd19-e844499743c0
72742025-04-30T06:46:18.590ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
72752025-04-30T06:46:18.590ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72762025-04-30T06:46:18.590ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55018 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f6a862d0-5bc6-4f75-bd19-e844499743c0
72772025-04-30T06:46:18.591ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4443, tv_nsec: 390500507 } })
72782025-04-30T06:46:18.591ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
72792025-04-30T06:46:18.592ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 31 a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 1...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49535 req_id = 8a5c19f0-fb04-43b7-81b0-d74c1464f007
72802025-04-30T06:46:18.592ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12713,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49535 req_id = 8a5c19f0-fb04-43b7-81b0-d74c1464f007
72812025-04-30T06:46:18.592ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72822025-04-30T06:46:18.592ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49535 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8a5c19f0-fb04-43b7-81b0-d74c1464f007
72832025-04-30T06:46:18.648ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
72842025-04-30T06:46:18.649ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72852025-04-30T06:46:18.650ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72862025-04-30T06:46:18.652ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 25 8a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c %............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58647 req_id = 36d856d9-7717-4026-815a-cb259001000c
72872025-04-30T06:46:18.652ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9610,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58647 req_id = 36d856d9-7717-4026-815a-cb259001000c
72882025-04-30T06:46:18.652ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
72892025-04-30T06:46:18.652ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9610, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58647 req_id = 36d856d9-7717-4026-815a-cb259001000c
72902025-04-30T06:46:18.658ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
72912025-04-30T06:46:18.658ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
72922025-04-30T06:46:18.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
72932025-04-30T06:46:18.659ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72942025-04-30T06:46:18.660ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72952025-04-30T06:46:18.660ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72962025-04-30T06:46:18.660ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
72972025-04-30T06:46:18.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9ad54b-e014-426d-868a-f0d561133f33", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
72982025-04-30T06:46:18.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
72992025-04-30T06:46:18.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b3e3cb-1ce8-4159-8888-0eaddc5a9262", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73002025-04-30T06:46:18.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
73012025-04-30T06:46:18.674ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
73022025-04-30T06:46:18.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb17f2a3-b500-4f4d-9196-eb3bfd6dc033", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73032025-04-30T06:46:18.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
73042025-04-30T06:46:18.679ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "082ca942-a3d7-4f44-adef-e53b0608d3fa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73052025-04-30T06:46:18.679ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
73062025-04-30T06:46:18.683ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c2fcd49-1c20-45d6-9fc8-64919b18541c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73072025-04-30T06:46:18.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
73082025-04-30T06:46:18.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b1cf215-f43d-4acd-a306-18c70c537e2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73092025-04-30T06:46:18.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
73102025-04-30T06:46:18.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e6f002-de13-4a1a-9b37-506765260676", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73112025-04-30T06:46:18.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
73122025-04-30T06:46:18.700ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23230ee6-37db-451e-9272-99a2889062e9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73132025-04-30T06:46:18.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
73142025-04-30T06:46:18.705ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d09e9be7-f56a-4fbe-9345-55ec8c24a63d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73152025-04-30T06:46:18.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
73162025-04-30T06:46:18.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af8263c2-4ce5-4a3a-9f64-a26430aa0628", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73172025-04-30T06:46:18.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
73182025-04-30T06:46:18.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de10153d-229e-48f5-ab6f-4adaf80b3fb9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73192025-04-30T06:46:18.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
73202025-04-30T06:46:18.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a051f3-6b48-494c-a09a-8d82b53f8090", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73212025-04-30T06:46:18.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
73222025-04-30T06:46:18.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9622f05-fa95-4716-9497-925f9cb189af", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73232025-04-30T06:46:18.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
73242025-04-30T06:46:18.715ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cabd4c1-1b09-4556-87e2-76e57e1f6a3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73252025-04-30T06:46:18.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
73262025-04-30T06:46:18.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dab5a2cf-8339-4903-aee9-98143cdb3039", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73272025-04-30T06:46:18.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
73282025-04-30T06:46:18.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d85305d6-44e8-4ca1-b7a6-9a1bb37418c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73292025-04-30T06:46:18.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
73302025-04-30T06:46:18.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82c712e8-992f-42d8-93c4-282027b55e12", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73312025-04-30T06:46:18.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
73322025-04-30T06:46:18.725ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73332025-04-30T06:46:18.725ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73342025-04-30T06:46:18.725ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73352025-04-30T06:46:18.725ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73362025-04-30T06:46:18.726ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73372025-04-30T06:46:18.729ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73382025-04-30T06:46:18.736ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73392025-04-30T06:46:18.736ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73402025-04-30T06:46:18.736ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73412025-04-30T06:46:18.736ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73422025-04-30T06:46:18.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58f8788c-5ec4-4eb6-9b58-be6ec6e5d272", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73432025-04-30T06:46:18.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
73442025-04-30T06:46:18.743ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73452025-04-30T06:46:18.744ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73462025-04-30T06:46:18.745ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73472025-04-30T06:46:18.752ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a02efec-7419-4d74-badb-fab588a1cf20", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73482025-04-30T06:46:18.752ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
73492025-04-30T06:46:18.761ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "870d3e67-0f62-4e7b-8e10-3284cb28458e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73502025-04-30T06:46:18.761ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
73512025-04-30T06:46:18.764ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c3c891d-f940-46b3-85a9-d468bf85cea1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73522025-04-30T06:46:18.764ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
73532025-04-30T06:46:18.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a14e95-07f9-4281-8729-9aada7b6b453", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73542025-04-30T06:46:18.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
73552025-04-30T06:46:18.780ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60105970-44a5-41b3-9172-dea6335c34e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73562025-04-30T06:46:18.780ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
73572025-04-30T06:46:18.782ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90f93ea-68f1-45f2-b0a8-b0b2c46698ef", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73582025-04-30T06:46:18.782ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
73592025-04-30T06:46:18.786ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98cb0ff2-56d9-48d7-8194-ea64478e3df8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73602025-04-30T06:46:18.786ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
73612025-04-30T06:46:18.790ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "298556d5-80df-45c7-a664-5e6425251f6e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73622025-04-30T06:46:18.790ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
73632025-04-30T06:46:18.799ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
73642025-04-30T06:46:18.804ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f46d6365-6bc2-48eb-8e70-5f23781268a3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73652025-04-30T06:46:18.804ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
73662025-04-30T06:46:18.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31e400d0-d2a7-495f-8e0f-680286320827", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73672025-04-30T06:46:18.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
73682025-04-30T06:46:18.824ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d786a32-17e9-43a5-a94e-14fd84fafd76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73692025-04-30T06:46:18.824ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
73702025-04-30T06:46:18.832ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf6c265-6c53-4fe5-b37f-ec11120616cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73712025-04-30T06:46:18.833ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
73722025-04-30T06:46:18.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04358550-81e5-4fd0-8592-a83413570bdb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73732025-04-30T06:46:18.835ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
73742025-04-30T06:46:18.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51524e00-fe6a-40cf-8bf8-9d12217369eb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73752025-04-30T06:46:18.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
73762025-04-30T06:46:18.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5009a1e9-9083-40d9-9d07-9bed94324072", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73772025-04-30T06:46:18.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
73782025-04-30T06:46:18.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "175b5ad2-8f3d-41a7-93f1-cbf423f42ffa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73792025-04-30T06:46:18.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
73802025-04-30T06:46:18.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8580a9ee-5de2-4f49-8e2c-81e51b8b5ddc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73812025-04-30T06:46:18.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
73822025-04-30T06:46:18.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e254d9-f40b-451e-b078-dc50ad6561a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73832025-04-30T06:46:18.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
73842025-04-30T06:46:18.850ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73852025-04-30T06:46:18.851ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73862025-04-30T06:46:18.851ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73872025-04-30T06:46:18.851ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73882025-04-30T06:46:18.858ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73892025-04-30T06:46:18.862ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73902025-04-30T06:46:18.870ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73912025-04-30T06:46:18.870ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73922025-04-30T06:46:18.870ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73932025-04-30T06:46:18.871ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73942025-04-30T06:46:18.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "312802f1-7002-40c3-89c9-6b9478b744e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
73952025-04-30T06:46:18.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
73962025-04-30T06:46:18.873ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73972025-04-30T06:46:18.874ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
73982025-04-30T06:46:18.874ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73992025-04-30T06:46:18.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5fff77-3c5b-49bc-a39a-66f362cc1048", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74002025-04-30T06:46:18.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
74012025-04-30T06:46:18.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa25a13d-6641-4b73-9bd7-f55f275d9a5c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74022025-04-30T06:46:18.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
74032025-04-30T06:46:18.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80df513-6542-4220-8c49-cce1a9e99987", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74042025-04-30T06:46:18.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
74052025-04-30T06:46:18.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a0f82f-e185-49f6-b0d8-0c089f90d3c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74062025-04-30T06:46:18.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
74072025-04-30T06:46:18.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d70c591-332c-4e56-8f12-4ae6b6bc3d6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74082025-04-30T06:46:18.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
74092025-04-30T06:46:18.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32cfa3c3-aac4-4159-a2e4-2fb2da22bf03", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:17 GMT"} })
74102025-04-30T06:46:18.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
74112025-04-30T06:46:18.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ed31cac-8fca-4408-b506-b1a9468c2f52", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
74122025-04-30T06:46:18.922ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
74132025-04-30T06:46:18.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6927d10-fe16-4b0e-a820-8fa2577d8b5c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
74142025-04-30T06:46:18.931ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74152025-04-30T06:46:18.931ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74162025-04-30T06:46:18.931ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74172025-04-30T06:46:18.931ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74182025-04-30T06:46:18.932ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74192025-04-30T06:46:18.938ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
74202025-04-30T06:46:18.939ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74212025-04-30T06:46:18.943ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74222025-04-30T06:46:18.943ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74232025-04-30T06:46:18.943ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74242025-04-30T06:46:18.943ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74252025-04-30T06:46:18.943ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74262025-04-30T06:46:18.944ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74272025-04-30T06:46:18.947ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74282025-04-30T06:46:18.952ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74292025-04-30T06:46:18.952ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74302025-04-30T06:46:18.952ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74312025-04-30T06:46:18.953ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74322025-04-30T06:46:19.006ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
74332025-04-30T06:46:19.006ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
74342025-04-30T06:46:19.006ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74352025-04-30T06:46:19.006ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74362025-04-30T06:46:19.006ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74372025-04-30T06:46:19.006ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
74382025-04-30T06:46:19.008ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74392025-04-30T06:46:19.008ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:58931 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
74402025-04-30T06:46:19.008ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74412025-04-30T06:46:19.008ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74422025-04-30T06:46:19.008ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
74432025-04-30T06:46:19.009ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
74442025-04-30T06:46:19.010ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:58931 req_id = 30e880f3-a798-4a61-8dd5-da70bc85665b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74452025-04-30T06:46:19.010ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 499 local_addr = [::1]:54590 method = GET remote_addr = [::1]:58931 req_id = 30e880f3-a798-4a61-8dd5-da70bc85665b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74462025-04-30T06:46:19.011ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "30e880f3-a798-4a61-8dd5-da70bc85665b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
74472025-04-30T06:46:19.013ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74482025-04-30T06:46:19.013ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74492025-04-30T06:46:19.013ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
74502025-04-30T06:46:19.015ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74512025-04-30T06:46:19.015ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74522025-04-30T06:46:19.015ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74532025-04-30T06:46:19.015ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74542025-04-30T06:46:19.016ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74552025-04-30T06:46:19.016ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74562025-04-30T06:46:19.024ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74572025-04-30T06:46:19.025ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74582025-04-30T06:46:19.025ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
74592025-04-30T06:46:19.025ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74602025-04-30T06:46:19.033ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
74612025-04-30T06:46:19.033ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
74622025-04-30T06:46:19.034ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74632025-04-30T06:46:19.038ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74642025-04-30T06:46:19.038ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74652025-04-30T06:46:19.038ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74662025-04-30T06:46:19.039ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74672025-04-30T06:46:19.043ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74682025-04-30T06:46:19.043ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
74692025-04-30T06:46:19.044ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74702025-04-30T06:46:19.044ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
74712025-04-30T06:46:19.044ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
74722025-04-30T06:46:19.089ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
74732025-04-30T06:46:19.089ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
74742025-04-30T06:46:19.089ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 538.037645ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
74752025-04-30T06:46:19.090ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
74762025-04-30T06:46:19.090ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74772025-04-30T06:46:19.090ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74782025-04-30T06:46:19.090ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74792025-04-30T06:46:19.090ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74802025-04-30T06:46:19.101ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74812025-04-30T06:46:19.104ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74822025-04-30T06:46:19.108ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74832025-04-30T06:46:19.109ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74842025-04-30T06:46:19.109ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74852025-04-30T06:46:19.110ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74862025-04-30T06:46:19.114ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74872025-04-30T06:46:19.116ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
74882025-04-30T06:46:19.132ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
74892025-04-30T06:46:19.132ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
74902025-04-30T06:46:19.134ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b1 58 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .X..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55409 req_id = d45b9592-4be3-4e29-8570-3bb89cb66bb9
74912025-04-30T06:46:19.134ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45400,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55409 req_id = d45b9592-4be3-4e29-8570-3bb89cb66bb9
74922025-04-30T06:46:19.135ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
74932025-04-30T06:46:19.135ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74942025-04-30T06:46:19.135ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55409 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d45b9592-4be3-4e29-8570-3bb89cb66bb9
74952025-04-30T06:46:19.136ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4443, tv_nsec: 935149925 } })
74962025-04-30T06:46:19.136ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
74972025-04-30T06:46:19.137ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f2 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .s..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56782 req_id = f7c4d08f-6e60-45c5-8fb5-5158a9bd5af7
74982025-04-30T06:46:19.137ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62067,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56782 req_id = f7c4d08f-6e60-45c5-8fb5-5158a9bd5af7
74992025-04-30T06:46:19.138ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75002025-04-30T06:46:19.138ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56782 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f7c4d08f-6e60-45c5-8fb5-5158a9bd5af7
75012025-04-30T06:46:19.245ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
75022025-04-30T06:46:19.245ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75032025-04-30T06:46:19.245ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75042025-04-30T06:46:19.246ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75052025-04-30T06:46:19.246ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
75062025-04-30T06:46:19.247ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75072025-04-30T06:46:19.247ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a2 46 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .F..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50462 req_id = 90b53042-f0ac-4d9d-8aa6-13d8ff563394
75082025-04-30T06:46:19.247ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41542,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50462 req_id = 90b53042-f0ac-4d9d-8aa6-13d8ff563394
75092025-04-30T06:46:19.248ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
75102025-04-30T06:46:19.248ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41542, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50462 req_id = 90b53042-f0ac-4d9d-8aa6-13d8ff563394
75112025-04-30T06:46:19.248ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
75122025-04-30T06:46:19.248ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
75132025-04-30T06:46:19.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
75142025-04-30T06:46:19.252ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75152025-04-30T06:46:19.257ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75162025-04-30T06:46:19.258ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75172025-04-30T06:46:19.258ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
75182025-04-30T06:46:19.258ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75192025-04-30T06:46:19.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e404f331-679e-4ae4-98e6-9922a46ed8d4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75202025-04-30T06:46:19.277ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
75212025-04-30T06:46:19.279ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75222025-04-30T06:46:19.280ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
75232025-04-30T06:46:19.280ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75242025-04-30T06:46:19.281ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1621656c-9613-4bc4-9e4d-d6f8f5381d2f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75252025-04-30T06:46:19.281ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
75262025-04-30T06:46:19.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e362eb86-c300-41d2-91ef-093dbbc49eee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75272025-04-30T06:46:19.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
75282025-04-30T06:46:19.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d8fd278-b305-4920-85de-0161080ca258", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75292025-04-30T06:46:19.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
75302025-04-30T06:46:19.300ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3243d891-8527-40ab-bbfb-62735cbbcc58", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75312025-04-30T06:46:19.300ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
75322025-04-30T06:46:19.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca54043e-3379-40a0-8485-fbc36f305f02", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75332025-04-30T06:46:19.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
75342025-04-30T06:46:19.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bb5b02f-e4a8-44fb-8d3e-0cd6704d535b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75352025-04-30T06:46:19.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
75362025-04-30T06:46:19.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a1864d-0ab4-4f3f-8801-268cd698eeac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75372025-04-30T06:46:19.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
75382025-04-30T06:46:19.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ceac388-e2e0-4ea5-9bf0-3d09e4bcff98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75392025-04-30T06:46:19.327ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
75402025-04-30T06:46:19.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eea2a01-1244-4adb-8c1b-c7541db62aad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75412025-04-30T06:46:19.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
75422025-04-30T06:46:19.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c130cc-f76d-4904-8be4-9ca8d0bfd8da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75432025-04-30T06:46:19.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
75442025-04-30T06:46:19.331ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
75452025-04-30T06:46:19.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4b904b3-b8c7-49a3-beb8-6af09e4e5a04", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75462025-04-30T06:46:19.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
75472025-04-30T06:46:19.334ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2d29fed-4a55-4a35-be81-7b71371cdfa3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75482025-04-30T06:46:19.334ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
75492025-04-30T06:46:19.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8990374a-128a-4c0a-b73e-e8c55e8b827e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75502025-04-30T06:46:19.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
75512025-04-30T06:46:19.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d95afcd6-da99-43d6-a251-bbff3c90f46e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75522025-04-30T06:46:19.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
75532025-04-30T06:46:19.340ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7327f570-575e-4e85-8960-fe5c4cb6abee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75542025-04-30T06:46:19.340ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
75552025-04-30T06:46:19.343ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58e6639d-5b4b-4df0-abec-bccbd6d3527d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75562025-04-30T06:46:19.343ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
75572025-04-30T06:46:19.345ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "725f5b95-1ca9-4caa-89af-246872bbd129", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75582025-04-30T06:46:19.345ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
75592025-04-30T06:46:19.347ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3d0264a-0ff1-4f03-b608-64db45a7914e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75602025-04-30T06:46:19.347ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
75612025-04-30T06:46:19.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db7bcde7-64e5-49ca-b7a7-ef5b5cfe701e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75622025-04-30T06:46:19.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
75632025-04-30T06:46:19.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7273b14-055a-40c6-a79a-cd510e854b89", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75642025-04-30T06:46:19.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
75652025-04-30T06:46:19.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e94145-ec54-4f3d-ac5d-f8fa4ad5061f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75662025-04-30T06:46:19.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
75672025-04-30T06:46:19.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8ae6f46-0659-4194-a1a3-36d6ea899a18", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75682025-04-30T06:46:19.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
75692025-04-30T06:46:19.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6149d26-a93d-4ca5-a572-7e42d7dad642", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75702025-04-30T06:46:19.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
75712025-04-30T06:46:19.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "847540e4-56cc-4e20-9f13-93e911cc9051", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75722025-04-30T06:46:19.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
75732025-04-30T06:46:19.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af55f4ab-5e71-4d10-b22e-b2f4d5520350", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75742025-04-30T06:46:19.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
75752025-04-30T06:46:19.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd5da4cd-8322-4f57-8015-b43df3653db7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75762025-04-30T06:46:19.359ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
75772025-04-30T06:46:19.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d031c4fe-bfc5-435b-8a92-6522a24cbd8f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75782025-04-30T06:46:19.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
75792025-04-30T06:46:19.363ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9537dc69-b39f-483d-ac5c-8feebbba9882", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75802025-04-30T06:46:19.363ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
75812025-04-30T06:46:19.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8940ab20-3426-48f8-bb9f-6e8e571e2f4c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75822025-04-30T06:46:19.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
75832025-04-30T06:46:19.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc8d32e7-e354-4b5b-b861-ddd421077cc8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75842025-04-30T06:46:19.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
75852025-04-30T06:46:19.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95edae3c-b428-410a-b66f-518ea1e10360", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75862025-04-30T06:46:19.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
75872025-04-30T06:46:19.374ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d8bc6d1-68d7-4ba7-9519-470afc57387c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75882025-04-30T06:46:19.374ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
75892025-04-30T06:46:19.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b54827-7efa-4525-b9a4-c5cbbd6f960c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75902025-04-30T06:46:19.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
75912025-04-30T06:46:19.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e892b0ed-98f1-4cf0-8112-9c5070f413f5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75922025-04-30T06:46:19.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
75932025-04-30T06:46:19.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eec64ad7-3353-4c1e-a0ea-d5f765fbac52", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75942025-04-30T06:46:19.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
75952025-04-30T06:46:19.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c973725c-b055-485f-9d83-fe6d2348a112", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
75962025-04-30T06:46:19.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
75972025-04-30T06:46:19.382ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75982025-04-30T06:46:19.382ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75992025-04-30T06:46:19.382ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76002025-04-30T06:46:19.382ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
76012025-04-30T06:46:19.394ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76022025-04-30T06:46:19.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4855e28-be8b-48d5-ac7f-3bedf1ba2ea8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76032025-04-30T06:46:19.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
76042025-04-30T06:46:19.398ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76052025-04-30T06:46:19.406ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76062025-04-30T06:46:19.406ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76072025-04-30T06:46:19.406ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
76082025-04-30T06:46:19.406ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76092025-04-30T06:46:19.407ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd33ecb3-0952-46ca-9733-3430e311d765", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76102025-04-30T06:46:19.407ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
76112025-04-30T06:46:19.407ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76122025-04-30T06:46:19.407ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
76132025-04-30T06:46:19.407ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76142025-04-30T06:46:19.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "651c7031-3395-43f8-b072-60b351e47cfa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76152025-04-30T06:46:19.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
76162025-04-30T06:46:19.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5ac9d8f-d726-4c40-860e-c90e27c373f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76172025-04-30T06:46:19.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
76182025-04-30T06:46:19.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6927902-1f04-4905-80c3-cb208f0257db", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76192025-04-30T06:46:19.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
76202025-04-30T06:46:19.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f3b457-0644-42ae-bbb8-e23e64adea43", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76212025-04-30T06:46:19.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
76222025-04-30T06:46:19.538ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
76232025-04-30T06:46:19.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf102d7d-ca99-402b-a5aa-dd95c097e8f2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76242025-04-30T06:46:19.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
76252025-04-30T06:46:19.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4176cd-9c70-4b00-9905-4b7ddb0b454a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76262025-04-30T06:46:19.545ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76272025-04-30T06:46:19.545ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76282025-04-30T06:46:19.545ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76292025-04-30T06:46:19.545ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76302025-04-30T06:46:19.546ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76312025-04-30T06:46:19.549ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
76322025-04-30T06:46:19.550ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76332025-04-30T06:46:19.551ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76342025-04-30T06:46:19.556ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76352025-04-30T06:46:19.556ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76362025-04-30T06:46:19.556ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76372025-04-30T06:46:19.556ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76382025-04-30T06:46:19.557ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76392025-04-30T06:46:19.557ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76402025-04-30T06:46:19.558ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 21b5b6bb-853f-4718-9dc5-642ee8a5d82d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76412025-04-30T06:46:19.558ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 21b5b6bb-853f-4718-9dc5-642ee8a5d82d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76422025-04-30T06:46:19.558ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 21b5b6bb-853f-4718-9dc5-642ee8a5d82d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76432025-04-30T06:46:19.559ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 21b5b6bb-853f-4718-9dc5-642ee8a5d82d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76442025-04-30T06:46:19.561ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76452025-04-30T06:46:19.571ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76462025-04-30T06:46:19.571ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76472025-04-30T06:46:19.571ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76482025-04-30T06:46:19.572ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76492025-04-30T06:46:19.572ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 14003 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 21b5b6bb-853f-4718-9dc5-642ee8a5d82d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
76502025-04-30T06:46:19.572ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76512025-04-30T06:46:19.575ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b5b6bb-853f-4718-9dc5-642ee8a5d82d", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76522025-04-30T06:46:19.575ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76532025-04-30T06:46:19.575ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76542025-04-30T06:46:19.575ZINFOtest_multiple_snapshots (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:51104 remote_addr = 127.0.0.1:62425
76552025-04-30T06:46:19.575ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0f833d89-a8b1-4d94-8a47-673f50cf9f68 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76562025-04-30T06:46:19.575ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0f833d89-a8b1-4d94-8a47-673f50cf9f68 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76572025-04-30T06:46:19.575ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0f833d89-a8b1-4d94-8a47-673f50cf9f68 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76582025-04-30T06:46:19.575ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0f833d89-a8b1-4d94-8a47-673f50cf9f68 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76592025-04-30T06:46:19.634ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
76602025-04-30T06:46:19.634ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
76612025-04-30T06:46:19.637ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 62505 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 0f833d89-a8b1-4d94-8a47-673f50cf9f68 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
76622025-04-30T06:46:19.637ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76632025-04-30T06:46:19.637ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76642025-04-30T06:46:19.637ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76652025-04-30T06:46:19.637ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
76662025-04-30T06:46:19.637ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76672025-04-30T06:46:19.638ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:59875 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76682025-04-30T06:46:19.639ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = 127.0.0.1:51104 method = GET remote_addr = 127.0.0.1:62425 req_id = b0d84786-5566-44c3-9d36-04a3b4c704aa uri = /3ae6adf1-675a-4c6a-a06e-473e2878df6e
76692025-04-30T06:46:19.639ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76702025-04-30T06:46:19.639ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f833d89-a8b1-4d94-8a47-673f50cf9f68", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76712025-04-30T06:46:19.639ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76722025-04-30T06:46:19.639ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76732025-04-30T06:46:19.639ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
76742025-04-30T06:46:19.644ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76752025-04-30T06:46:19.644ZINFOtest_multiple_snapshots (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 6374 local_addr = 127.0.0.1:51104 method = GET remote_addr = 127.0.0.1:62425 req_id = b0d84786-5566-44c3-9d36-04a3b4c704aa response_code = 200 uri = /3ae6adf1-675a-4c6a-a06e-473e2878df6e
76762025-04-30T06:46:19.645ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:59875 req_id = 330d602f-11b2-492e-b653-19c440ad9e62 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76772025-04-30T06:46:19.647ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76782025-04-30T06:46:19.648ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2515 local_addr = [::1]:54590 method = GET remote_addr = [::1]:59875 req_id = 330d602f-11b2-492e-b653-19c440ad9e62 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76792025-04-30T06:46:19.662ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76802025-04-30T06:46:19.662ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76812025-04-30T06:46:19.662ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
76822025-04-30T06:46:19.662ZDEBGtest_multiple_snapshots (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76832025-04-30T06:46:19.662ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76842025-04-30T06:46:19.662ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
76852025-04-30T06:46:19.663ZDEBGtest_multiple_snapshots (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
76862025-04-30T06:46:19.663ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
76872025-04-30T06:46:19.668ZTRACtest_multiple_snapshots (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
76882025-04-30T06:46:19.668ZDEBGtest_multiple_snapshots (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:46:19.663413648Z } timeseries_name = http_service:request_latency_histogram
76892025-04-30T06:46:19.681ZDEBGtest_multiple_snapshots (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
76902025-04-30T06:46:19.681ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
76912025-04-30T06:46:19.681ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76922025-04-30T06:46:19.682ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76932025-04-30T06:46:19.682ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76942025-04-30T06:46:19.682ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76952025-04-30T06:46:19.682ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "330d602f-11b2-492e-b653-19c440ad9e62", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
76962025-04-30T06:46:19.685ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76972025-04-30T06:46:19.685ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76982025-04-30T06:46:19.685ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
76992025-04-30T06:46:19.687ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77002025-04-30T06:46:19.687ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77012025-04-30T06:46:19.687ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77022025-04-30T06:46:19.687ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77032025-04-30T06:46:19.689ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77042025-04-30T06:46:19.690ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77052025-04-30T06:46:19.697ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77062025-04-30T06:46:19.697ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77072025-04-30T06:46:19.697ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
77082025-04-30T06:46:19.697ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77092025-04-30T06:46:19.698ZTRACtest_multiple_snapshots (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77102025-04-30T06:46:19.698ZDEBGtest_multiple_snapshots (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:46:19.681036150Z } timeseries_name = collection_target:cpus_provisioned
77112025-04-30T06:46:19.700ZDEBGtest_multiple_snapshots (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77122025-04-30T06:46:19.700ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77132025-04-30T06:46:19.706ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77142025-04-30T06:46:19.736ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77152025-04-30T06:46:19.736ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77162025-04-30T06:46:19.736ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77172025-04-30T06:46:19.737ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77182025-04-30T06:46:19.739ZTRACtest_multiple_snapshots (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77192025-04-30T06:46:19.739ZDEBGtest_multiple_snapshots (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:46:19.700250503Z } timeseries_name = collection_target:ram_provisioned
77202025-04-30T06:46:19.740ZDEBGtest_multiple_snapshots (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77212025-04-30T06:46:19.740ZTRACtest_multiple_snapshots (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
77222025-04-30T06:46:19.744ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77232025-04-30T06:46:19.744ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
77242025-04-30T06:46:19.746ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77252025-04-30T06:46:19.747ZTRACtest_multiple_snapshots (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77262025-04-30T06:46:19.747ZDEBGtest_multiple_snapshots (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:46:19.740046524Z } timeseries_name = collection_target:virtual_disk_space_provisioned
77272025-04-30T06:46:19.753ZDEBGtest_multiple_snapshots (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
77282025-04-30T06:46:19.754ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
77292025-04-30T06:46:19.763ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
77302025-04-30T06:46:19.767ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
77312025-04-30T06:46:19.771ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 10 table_name = fields_string
77322025-04-30T06:46:19.771ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
77332025-04-30T06:46:19.775ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 5 table_name = fields_u16
77342025-04-30T06:46:19.776ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
77352025-04-30T06:46:19.780ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 17 table_name = fields_uuid
77362025-04-30T06:46:19.780ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
77372025-04-30T06:46:19.786ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 5 table_name = measurements_histogramu64
77382025-04-30T06:46:19.786ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
77392025-04-30T06:46:19.791ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 12 table_name = measurements_i64
77402025-04-30T06:46:19.791ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77412025-04-30T06:46:19.814ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77422025-04-30T06:46:19.814ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77432025-04-30T06:46:19.815ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 724.980106ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77442025-04-30T06:46:19.815ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77452025-04-30T06:46:19.815ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77462025-04-30T06:46:19.815ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77472025-04-30T06:46:19.815ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77482025-04-30T06:46:19.816ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77492025-04-30T06:46:19.818ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77502025-04-30T06:46:19.837ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77512025-04-30T06:46:19.843ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77522025-04-30T06:46:19.843ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77532025-04-30T06:46:19.843ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77542025-04-30T06:46:19.844ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77552025-04-30T06:46:19.845ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
77562025-04-30T06:46:19.852ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
77572025-04-30T06:46:19.853ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77582025-04-30T06:46:19.853ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 41 41 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 AA..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53484 req_id = e3e63837-e79d-4ac8-a3bd-0a5ce3d61d71
77592025-04-30T06:46:19.853ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16705,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53484 req_id = e3e63837-e79d-4ac8-a3bd-0a5ce3d61d71
77602025-04-30T06:46:19.854ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
77612025-04-30T06:46:19.854ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77622025-04-30T06:46:19.854ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53484 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e3e63837-e79d-4ac8-a3bd-0a5ce3d61d71
77632025-04-30T06:46:19.855ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4444, tv_nsec: 654360114 } })
77642025-04-30T06:46:19.855ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
77652025-04-30T06:46:19.856ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ac 58 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .X..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58060 req_id = f0809c64-29c1-4117-92c1-c6f3291b808b
77662025-04-30T06:46:19.856ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44120,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58060 req_id = f0809c64-29c1-4117-92c1-c6f3291b808b
77672025-04-30T06:46:19.856ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77682025-04-30T06:46:19.856ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58060 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f0809c64-29c1-4117-92c1-c6f3291b808b
77692025-04-30T06:46:19.909ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
77702025-04-30T06:46:19.910ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77712025-04-30T06:46:19.910ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77722025-04-30T06:46:19.910ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77732025-04-30T06:46:19.910ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
77742025-04-30T06:46:19.911ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77752025-04-30T06:46:19.911ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1e 39 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .9..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52914 req_id = 8116c000-ba9e-4aea-8e7b-5ee0026fddc1
77762025-04-30T06:46:19.911ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7737,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52914 req_id = 8116c000-ba9e-4aea-8e7b-5ee0026fddc1
77772025-04-30T06:46:19.911ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
77782025-04-30T06:46:19.912ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7737, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52914 req_id = 8116c000-ba9e-4aea-8e7b-5ee0026fddc1
77792025-04-30T06:46:19.912ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
77802025-04-30T06:46:19.912ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
77812025-04-30T06:46:19.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
77822025-04-30T06:46:19.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3baa0dc-c927-46d2-9015-e73cebf91d08", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:18 GMT"} })
77832025-04-30T06:46:19.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
77842025-04-30T06:46:19.915ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77852025-04-30T06:46:19.921ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77862025-04-30T06:46:19.921ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77872025-04-30T06:46:19.921ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
77882025-04-30T06:46:19.922ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77892025-04-30T06:46:19.926ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f47034-d132-4a61-a83c-645c14f85e27", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
77902025-04-30T06:46:19.926ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
77912025-04-30T06:46:19.926ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77922025-04-30T06:46:19.926ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
77932025-04-30T06:46:19.927ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77942025-04-30T06:46:19.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ac35a90-4b9f-4506-8202-44045ef9ffd8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
77952025-04-30T06:46:19.930ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
77962025-04-30T06:46:19.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "946ed6ff-214e-4ec3-ab0d-4581bd61fab6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
77972025-04-30T06:46:19.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
77982025-04-30T06:46:19.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a20ca0cf-c6aa-440a-873e-bebfa4d86071", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
77992025-04-30T06:46:19.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
78002025-04-30T06:46:19.941ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b78f3f66-64cf-4094-b565-fce81afc426d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78012025-04-30T06:46:19.941ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
78022025-04-30T06:46:19.957ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9286df0-624e-433d-b5fd-e0f93068f1f0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78032025-04-30T06:46:19.958ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
78042025-04-30T06:46:19.960ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15c1e3ac-b1d1-4b7e-b2b8-c919325e3b57", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78052025-04-30T06:46:19.961ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
78062025-04-30T06:46:19.967ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad35c66c-7649-47a8-94ee-079076309cad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78072025-04-30T06:46:19.967ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
78082025-04-30T06:46:19.969ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a49d211-7495-454e-accf-f95c1c88d5fa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78092025-04-30T06:46:19.969ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
78102025-04-30T06:46:19.970ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4cb2254-22ea-4f67-b0f7-dbe2c4783b5b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78112025-04-30T06:46:19.970ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
78122025-04-30T06:46:19.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "177d7627-f963-428f-9568-a6264e5a5d43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78132025-04-30T06:46:19.972ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
78142025-04-30T06:46:19.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5d020c7-dfac-47d5-9315-3dfb5b6efdc8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78152025-04-30T06:46:19.978ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
78162025-04-30T06:46:19.979ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
78172025-04-30T06:46:19.980ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78182025-04-30T06:46:19.980ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78192025-04-30T06:46:19.983ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
78202025-04-30T06:46:19.989ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
78212025-04-30T06:46:19.994ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
78222025-04-30T06:46:19.994ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
78232025-04-30T06:46:19.997ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
78242025-04-30T06:46:19.997ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78252025-04-30T06:46:19.999ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7f00508-6345-48b7-99d8-805db888e456", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78262025-04-30T06:46:19.999ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
78272025-04-30T06:46:20.013ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d38a1ed-bfa3-4092-b3db-9699efb752cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78282025-04-30T06:46:20.013ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
78292025-04-30T06:46:20.017ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e54487d6-4c6e-48ed-8100-bf636692b456", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78302025-04-30T06:46:20.017ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
78312025-04-30T06:46:20.020ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "949e5574-9de7-4074-affa-889d5a4e06c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78322025-04-30T06:46:20.020ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
78332025-04-30T06:46:20.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f481ec0c-b6e0-4aa1-89fb-0407448e9496", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78342025-04-30T06:46:20.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
78352025-04-30T06:46:20.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e9104e-3d4b-4008-97ca-9960418e0776", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78362025-04-30T06:46:20.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
78372025-04-30T06:46:20.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "773c6e0d-0e0f-4095-a48a-05600b592ab2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78382025-04-30T06:46:20.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
78392025-04-30T06:46:20.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f949a20-8c00-4327-99a1-431b4a4b561a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78402025-04-30T06:46:20.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
78412025-04-30T06:46:20.030ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78422025-04-30T06:46:20.030ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78432025-04-30T06:46:20.030ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78442025-04-30T06:46:20.031ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
78452025-04-30T06:46:20.031ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78462025-04-30T06:46:20.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27158b10-b268-4356-b560-5f3f2626f2c5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78472025-04-30T06:46:20.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
78482025-04-30T06:46:20.035ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b047e806-09ab-4a43-9763-ab57cbe3f769", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78492025-04-30T06:46:20.035ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
78502025-04-30T06:46:20.035ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78512025-04-30T06:46:20.041ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78522025-04-30T06:46:20.041ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78532025-04-30T06:46:20.041ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
78542025-04-30T06:46:20.042ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78552025-04-30T06:46:20.045ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b23b44f-1130-459f-9b11-671874c01b88", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78562025-04-30T06:46:20.045ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
78572025-04-30T06:46:20.048ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78582025-04-30T06:46:20.048ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
78592025-04-30T06:46:20.049ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78602025-04-30T06:46:20.052ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "202b953b-11b7-460d-ad0b-b598ba1052a4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78612025-04-30T06:46:20.052ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
78622025-04-30T06:46:20.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5a5e82-37db-49dd-afda-bd44c0bf81e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78632025-04-30T06:46:20.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
78642025-04-30T06:46:20.065ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "428d1319-f1b2-4930-9177-9cf2c8c9a9b3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78652025-04-30T06:46:20.065ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
78662025-04-30T06:46:20.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e58405cd-3c89-49ec-857f-be89885530fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78672025-04-30T06:46:20.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
78682025-04-30T06:46:20.081ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4e4e8bb-f487-48e9-8109-e7d8d556e20c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78692025-04-30T06:46:20.081ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
78702025-04-30T06:46:20.082ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78712025-04-30T06:46:20.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "219e3d5d-1534-4f0b-86d2-b93fcf47973b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78722025-04-30T06:46:20.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
78732025-04-30T06:46:20.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a8f27fc-3442-4d28-97d7-373caa196238", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78742025-04-30T06:46:20.086ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
78752025-04-30T06:46:20.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0645262-d6d8-49b7-8de2-6666a1af901a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78762025-04-30T06:46:20.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
78772025-04-30T06:46:20.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52fe50ea-2f33-4aae-a3c2-6738bb887d07", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78782025-04-30T06:46:20.090ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
78792025-04-30T06:46:20.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98596e63-7831-4275-b61b-18071c33c05c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78802025-04-30T06:46:20.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
78812025-04-30T06:46:20.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "938841a8-b9c7-4fd9-8ac1-4bc21c4b0ef5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78822025-04-30T06:46:20.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
78832025-04-30T06:46:20.097ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ecb03a1-a3ef-4e8d-bda4-ca5400934e72", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78842025-04-30T06:46:20.097ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
78852025-04-30T06:46:20.102ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a408f875-5f6d-448e-be2f-d0a5a92e00b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78862025-04-30T06:46:20.103ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
78872025-04-30T06:46:20.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "560f52be-b87e-4ec5-a688-62e241630e32", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78882025-04-30T06:46:20.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
78892025-04-30T06:46:20.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "846ec0b0-8ce5-46be-a9a3-61aa350cd4f7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78902025-04-30T06:46:20.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
78912025-04-30T06:46:20.149ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3e360b7-202e-4a55-bea8-0d6d845b613b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78922025-04-30T06:46:20.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
78932025-04-30T06:46:20.152ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0613fa6d-8996-4dbf-ae69-b22bbcca2a4e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78942025-04-30T06:46:20.152ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
78952025-04-30T06:46:20.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc10bff8-db17-4d61-b55a-28cbc01ad7a3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78962025-04-30T06:46:20.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
78972025-04-30T06:46:20.155ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d32f790c-c90e-4e85-b996-c798e6a056f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
78982025-04-30T06:46:20.155ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
78992025-04-30T06:46:20.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2381d2a9-06c1-4cca-852f-a540fce53268", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
79002025-04-30T06:46:20.157ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
79012025-04-30T06:46:20.158ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
79022025-04-30T06:46:20.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fa7bb81-dd7d-4cb5-907c-169727fc0ced", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
79032025-04-30T06:46:20.171ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79042025-04-30T06:46:20.171ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79052025-04-30T06:46:20.171ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79062025-04-30T06:46:20.171ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79072025-04-30T06:46:20.178ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79082025-04-30T06:46:20.219ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79092025-04-30T06:46:20.219ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79102025-04-30T06:46:20.219ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79112025-04-30T06:46:20.219ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
79122025-04-30T06:46:20.221ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79132025-04-30T06:46:20.230ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79142025-04-30T06:46:20.237ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79152025-04-30T06:46:20.238ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79162025-04-30T06:46:20.238ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79172025-04-30T06:46:20.238ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79182025-04-30T06:46:20.238ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79192025-04-30T06:46:20.239ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79202025-04-30T06:46:20.253ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79212025-04-30T06:46:20.273ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79222025-04-30T06:46:20.273ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79232025-04-30T06:46:20.273ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
79242025-04-30T06:46:20.274ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79252025-04-30T06:46:20.275ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79262025-04-30T06:46:20.280ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79272025-04-30T06:46:20.280ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79282025-04-30T06:46:20.280ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79292025-04-30T06:46:20.281ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79302025-04-30T06:46:20.283ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79312025-04-30T06:46:20.283ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
79322025-04-30T06:46:20.284ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79332025-04-30T06:46:20.378ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
79342025-04-30T06:46:20.385ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
79352025-04-30T06:46:20.386ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:55833 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79362025-04-30T06:46:20.388ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:55833 req_id = 87a50ad0-3d83-4ea5-b2e0-10dcf80939f9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79372025-04-30T06:46:20.389ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 882 local_addr = [::1]:54590 method = GET remote_addr = [::1]:55833 req_id = 87a50ad0-3d83-4ea5-b2e0-10dcf80939f9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79382025-04-30T06:46:20.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a50ad0-3d83-4ea5-b2e0-10dcf80939f9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
79392025-04-30T06:46:20.393ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79402025-04-30T06:46:20.393ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79412025-04-30T06:46:20.393ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
79422025-04-30T06:46:20.395ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79432025-04-30T06:46:20.395ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79442025-04-30T06:46:20.395ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79452025-04-30T06:46:20.395ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79462025-04-30T06:46:20.396ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79472025-04-30T06:46:20.403ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79482025-04-30T06:46:20.426ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79492025-04-30T06:46:20.426ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79502025-04-30T06:46:20.426ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79512025-04-30T06:46:20.427ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79522025-04-30T06:46:20.515ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
79532025-04-30T06:46:20.523ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79542025-04-30T06:46:20.524ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79552025-04-30T06:46:20.524ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 709.038253ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79562025-04-30T06:46:20.524ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
79572025-04-30T06:46:20.524ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79582025-04-30T06:46:20.524ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79592025-04-30T06:46:20.524ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79602025-04-30T06:46:20.524ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79612025-04-30T06:46:20.525ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79622025-04-30T06:46:20.529ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79632025-04-30T06:46:20.535ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79642025-04-30T06:46:20.537ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79652025-04-30T06:46:20.537ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79662025-04-30T06:46:20.539ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79672025-04-30T06:46:20.548ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
79682025-04-30T06:46:20.548ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79692025-04-30T06:46:20.549ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 86 a8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51845 req_id = 5502afa0-99ef-48b9-a37e-13bd7d45088a
79702025-04-30T06:46:20.549ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34472,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51845 req_id = 5502afa0-99ef-48b9-a37e-13bd7d45088a
79712025-04-30T06:46:20.549ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
79722025-04-30T06:46:20.550ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79732025-04-30T06:46:20.550ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51845 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5502afa0-99ef-48b9-a37e-13bd7d45088a
79742025-04-30T06:46:20.550ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4445, tv_nsec: 349713010 } })
79752025-04-30T06:46:20.550ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
79762025-04-30T06:46:20.551ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 b5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62918 req_id = 1ebcc17b-0925-4477-8477-6057c0f27f44
79772025-04-30T06:46:20.551ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41653,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62918 req_id = 1ebcc17b-0925-4477-8477-6057c0f27f44
79782025-04-30T06:46:20.551ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79792025-04-30T06:46:20.551ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62918 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1ebcc17b-0925-4477-8477-6057c0f27f44
79802025-04-30T06:46:20.606ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
79812025-04-30T06:46:20.606ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79822025-04-30T06:46:20.606ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79832025-04-30T06:46:20.606ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79842025-04-30T06:46:20.606ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
79852025-04-30T06:46:20.607ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79862025-04-30T06:46:20.608ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0a 3e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .>..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54871 req_id = e5ca8c7f-f815-4971-8b9e-6f25c8b9d561
79872025-04-30T06:46:20.608ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2622,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54871 req_id = e5ca8c7f-f815-4971-8b9e-6f25c8b9d561
79882025-04-30T06:46:20.608ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
79892025-04-30T06:46:20.608ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54871 req_id = e5ca8c7f-f815-4971-8b9e-6f25c8b9d561
79902025-04-30T06:46:20.608ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
79912025-04-30T06:46:20.608ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
79922025-04-30T06:46:20.609ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
79932025-04-30T06:46:20.609ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79942025-04-30T06:46:20.616ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79952025-04-30T06:46:20.616ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79962025-04-30T06:46:20.616ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
79972025-04-30T06:46:20.617ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79982025-04-30T06:46:20.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb15beb-0975-4090-a89f-14191c0e2c3e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
79992025-04-30T06:46:20.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
80002025-04-30T06:46:20.620ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55eb2737-5ff3-4530-87f7-2037b99399e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80012025-04-30T06:46:20.620ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
80022025-04-30T06:46:20.620ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80032025-04-30T06:46:20.620ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
80042025-04-30T06:46:20.621ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80052025-04-30T06:46:20.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a9471e-6cd7-4392-9f2e-92dbecff237b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80062025-04-30T06:46:20.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
80072025-04-30T06:46:20.626ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12758f78-9a22-45c1-80d3-e6f77a0c8d4e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80082025-04-30T06:46:20.626ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
80092025-04-30T06:46:20.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85348779-df25-4c0a-b673-57c9fc20b468", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80102025-04-30T06:46:20.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
80112025-04-30T06:46:20.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80122025-04-30T06:46:20.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80132025-04-30T06:46:20.637ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d0fba12-c35b-461b-a33b-9ef42f436789", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80142025-04-30T06:46:20.637ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
80152025-04-30T06:46:20.637ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80162025-04-30T06:46:20.639ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80172025-04-30T06:46:20.639ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80182025-04-30T06:46:20.639ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80192025-04-30T06:46:20.644ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc15e765-c949-40a4-bcd3-248e98776f49", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80202025-04-30T06:46:20.644ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
80212025-04-30T06:46:20.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ae2edd6-8135-4ac6-927f-c47ce80d4210", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80222025-04-30T06:46:20.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
80232025-04-30T06:46:20.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6a5c715-4080-4cdd-8183-963f1cb71c82", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80242025-04-30T06:46:20.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
80252025-04-30T06:46:20.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c902bd8-4af6-4f95-b268-e7e16bec4a12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80262025-04-30T06:46:20.655ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
80272025-04-30T06:46:20.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5b3ee48-9508-413c-8f46-3b7133325222", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80282025-04-30T06:46:20.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
80292025-04-30T06:46:20.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf6a3cbc-4dbf-4479-8de8-dc68c5daea3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80302025-04-30T06:46:20.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
80312025-04-30T06:46:20.662ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
80322025-04-30T06:46:20.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac3b36c7-6910-487a-8753-12f523dfae85", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80332025-04-30T06:46:20.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
80342025-04-30T06:46:20.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "505a49c0-8e4d-4e9c-af0f-3404375f60ab", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80352025-04-30T06:46:20.681ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
80362025-04-30T06:46:20.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a312476d-098c-415a-8d8d-dc434beb9916", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80372025-04-30T06:46:20.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
80382025-04-30T06:46:20.691ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbbe83a4-044c-4e08-8a46-2efdd22e4185", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80392025-04-30T06:46:20.691ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
80402025-04-30T06:46:20.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2168b66e-e126-42a9-b992-bdd72ce617f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80412025-04-30T06:46:20.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
80422025-04-30T06:46:20.715ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80432025-04-30T06:46:20.715ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80442025-04-30T06:46:20.715ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80452025-04-30T06:46:20.715ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
80462025-04-30T06:46:20.716ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80472025-04-30T06:46:20.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa311aa3-5745-4b36-a9b5-721c942dce04", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80482025-04-30T06:46:20.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
80492025-04-30T06:46:20.720ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80502025-04-30T06:46:20.731ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80512025-04-30T06:46:20.731ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80522025-04-30T06:46:20.731ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
80532025-04-30T06:46:20.731ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80542025-04-30T06:46:20.734ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80552025-04-30T06:46:20.734ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
80562025-04-30T06:46:20.735ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80572025-04-30T06:46:20.735ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d2804ef-14c2-4fee-95d6-bf718271a60a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80582025-04-30T06:46:20.736ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
80592025-04-30T06:46:20.738ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8736205-5088-4176-a056-9d3cb615cb12", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80602025-04-30T06:46:20.738ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
80612025-04-30T06:46:20.739ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89888f11-933b-4fd4-b4ef-255289416c19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80622025-04-30T06:46:20.740ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
80632025-04-30T06:46:20.741ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6f9bd6-5bc5-4cbb-b099-d818c4d1d716", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80642025-04-30T06:46:20.743ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
80652025-04-30T06:46:20.745ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17084690-ba7f-4bc6-b9e5-01faf3a5280f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80662025-04-30T06:46:20.745ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
80672025-04-30T06:46:20.747ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "654ed292-8b39-4569-8223-dac01d75b4d9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80682025-04-30T06:46:20.747ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
80692025-04-30T06:46:20.751ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2bd84c0-213d-4306-9bab-e996a58eb0c3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80702025-04-30T06:46:20.752ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
80712025-04-30T06:46:20.756ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eaa24ad-a250-4a75-967a-a08e5204eac4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80722025-04-30T06:46:20.756ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
80732025-04-30T06:46:20.778ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c539847-8d23-494a-8324-2b8abac9f55b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80742025-04-30T06:46:20.778ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
80752025-04-30T06:46:20.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e30cf29-532e-4e88-9ddb-1bcc6ec52efa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80762025-04-30T06:46:20.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
80772025-04-30T06:46:20.785ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
80782025-04-30T06:46:20.791ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a132368-5aed-4d81-842d-e64d5362a413", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80792025-04-30T06:46:20.792ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
80802025-04-30T06:46:20.795ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27a653a8-9e88-4f0b-9194-a81266f5aa26", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80812025-04-30T06:46:20.796ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
80822025-04-30T06:46:20.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9665ff-b975-411a-8ea7-cbfaf258cf3a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80832025-04-30T06:46:20.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
80842025-04-30T06:46:20.799ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04f0e8da-558e-4f0f-aceb-4d1c80d4a01f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80852025-04-30T06:46:20.799ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
80862025-04-30T06:46:20.800ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2035f4b3-8531-40e2-a8da-be1c295535ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80872025-04-30T06:46:20.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
80882025-04-30T06:46:20.802ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "073a89af-48df-4fdb-ad01-e0a78ff04716", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80892025-04-30T06:46:20.802ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
80902025-04-30T06:46:20.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eaf5acd-6d70-4c95-a9a6-c72dfb659f3c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80912025-04-30T06:46:20.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
80922025-04-30T06:46:20.807ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d665ec11-e9b1-4356-8bd8-2e6f82206dba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80932025-04-30T06:46:20.807ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
80942025-04-30T06:46:20.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1ddcf5-64d0-4f78-852e-e7f694f3a319", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80952025-04-30T06:46:20.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
80962025-04-30T06:46:20.812ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7319df6-1996-4241-ab44-289018992055", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80972025-04-30T06:46:20.813ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
80982025-04-30T06:46:20.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "704d7dc6-fd89-4301-8992-a1c1e2eb010f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
80992025-04-30T06:46:20.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
81002025-04-30T06:46:20.817ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52104fac-4e35-41bc-8cbe-31e82605237d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81012025-04-30T06:46:20.817ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
81022025-04-30T06:46:20.823ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb78987-daca-4ca0-90e9-507b193f7ff1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81032025-04-30T06:46:20.823ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
81042025-04-30T06:46:20.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63778a16-eb6c-44e4-a8b2-084fd6e15bda", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81052025-04-30T06:46:20.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
81062025-04-30T06:46:20.831ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "892de70d-7e88-43e8-a8cb-6bc2d7fd9327", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81072025-04-30T06:46:20.831ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
81082025-04-30T06:46:20.847ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81092025-04-30T06:46:20.847ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81102025-04-30T06:46:20.847ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81112025-04-30T06:46:20.847ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81122025-04-30T06:46:20.847ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81132025-04-30T06:46:20.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edfcaf35-b1d5-425b-89b6-a77da36f5a81", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81142025-04-30T06:46:20.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
81152025-04-30T06:46:20.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9158a0b1-49ea-4f5a-bdcb-1090c4f945e4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:19 GMT"} })
81162025-04-30T06:46:20.852ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81172025-04-30T06:46:20.852ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81182025-04-30T06:46:20.852ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81192025-04-30T06:46:20.853ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81202025-04-30T06:46:20.853ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81212025-04-30T06:46:20.853ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81222025-04-30T06:46:20.876ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81232025-04-30T06:46:20.882ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81242025-04-30T06:46:20.882ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81252025-04-30T06:46:20.883ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81262025-04-30T06:46:20.887ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81272025-04-30T06:46:20.896ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81282025-04-30T06:46:20.896ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81292025-04-30T06:46:20.896ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81302025-04-30T06:46:20.896ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81312025-04-30T06:46:20.896ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81322025-04-30T06:46:20.897ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81332025-04-30T06:46:20.900ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81342025-04-30T06:46:20.901ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81352025-04-30T06:46:20.901ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81362025-04-30T06:46:20.902ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81372025-04-30T06:46:20.906ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81382025-04-30T06:46:20.907ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81392025-04-30T06:46:20.907ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81402025-04-30T06:46:20.908ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81412025-04-30T06:46:20.985ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81422025-04-30T06:46:20.985ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81432025-04-30T06:46:20.985ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
81442025-04-30T06:46:20.986ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
81452025-04-30T06:46:20.993ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
81462025-04-30T06:46:20.993ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
81472025-04-30T06:46:20.997ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
81482025-04-30T06:46:20.997ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81492025-04-30T06:46:21.094ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
81502025-04-30T06:46:21.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
81512025-04-30T06:46:21.097ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:42295 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81522025-04-30T06:46:21.097ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
81532025-04-30T06:46:21.098ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:42295 req_id = 10054f55-c248-4ab9-a21c-aa17e922c220 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81542025-04-30T06:46:21.098ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 886 local_addr = [::1]:54590 method = GET remote_addr = [::1]:42295 req_id = 10054f55-c248-4ab9-a21c-aa17e922c220 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81552025-04-30T06:46:21.100ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "10054f55-c248-4ab9-a21c-aa17e922c220", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
81562025-04-30T06:46:21.103ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81572025-04-30T06:46:21.103ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81582025-04-30T06:46:21.103ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
81592025-04-30T06:46:21.107ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81602025-04-30T06:46:21.107ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81612025-04-30T06:46:21.107ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81622025-04-30T06:46:21.107ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81632025-04-30T06:46:21.108ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81642025-04-30T06:46:21.129ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81652025-04-30T06:46:21.143ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81662025-04-30T06:46:21.143ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81672025-04-30T06:46:21.144ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81682025-04-30T06:46:21.145ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81692025-04-30T06:46:21.156ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
81702025-04-30T06:46:21.209ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81712025-04-30T06:46:21.209ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81722025-04-30T06:46:21.209ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81732025-04-30T06:46:21.209ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81742025-04-30T06:46:21.211ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81752025-04-30T06:46:21.222ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81762025-04-30T06:46:21.230ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81772025-04-30T06:46:21.230ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81782025-04-30T06:46:21.230ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81792025-04-30T06:46:21.231ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81802025-04-30T06:46:21.281ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81812025-04-30T06:46:21.281ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
81822025-04-30T06:46:21.282ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81832025-04-30T06:46:21.283ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81842025-04-30T06:46:21.283ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81852025-04-30T06:46:21.283ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 759.324264ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81862025-04-30T06:46:21.283ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
81872025-04-30T06:46:21.283ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81882025-04-30T06:46:21.283ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81892025-04-30T06:46:21.284ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81902025-04-30T06:46:21.284ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81912025-04-30T06:46:21.284ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81922025-04-30T06:46:21.290ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81932025-04-30T06:46:21.295ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81942025-04-30T06:46:21.295ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81952025-04-30T06:46:21.295ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81962025-04-30T06:46:21.296ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81972025-04-30T06:46:21.306ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
81982025-04-30T06:46:21.306ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
81992025-04-30T06:46:21.307ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 56 2b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 V+..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63125 req_id = 6edbf6ca-8e0c-4b89-af3f-40b835a7caff
82002025-04-30T06:46:21.307ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22059,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63125 req_id = 6edbf6ca-8e0c-4b89-af3f-40b835a7caff
82012025-04-30T06:46:21.307ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
82022025-04-30T06:46:21.307ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82032025-04-30T06:46:21.308ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63125 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6edbf6ca-8e0c-4b89-af3f-40b835a7caff
82042025-04-30T06:46:21.308ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4446, tv_nsec: 107596733 } })
82052025-04-30T06:46:21.308ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
82062025-04-30T06:46:21.309ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ae 36 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .6..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64275 req_id = 87394ad2-9e52-4420-9339-3ef3fbe38fd6
82072025-04-30T06:46:21.309ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44598,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64275 req_id = 87394ad2-9e52-4420-9339-3ef3fbe38fd6
82082025-04-30T06:46:21.309ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82092025-04-30T06:46:21.309ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64275 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 87394ad2-9e52-4420-9339-3ef3fbe38fd6
82102025-04-30T06:46:21.379ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82112025-04-30T06:46:21.380ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 19 29 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .)..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57068 req_id = 2a99213f-7d4f-4a6b-9618-c8dc78f7f933
82122025-04-30T06:46:21.380ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6441,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57068 req_id = 2a99213f-7d4f-4a6b-9618-c8dc78f7f933
82132025-04-30T06:46:21.380ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
82142025-04-30T06:46:21.380ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6441, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57068 req_id = 2a99213f-7d4f-4a6b-9618-c8dc78f7f933
82152025-04-30T06:46:21.380ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
82162025-04-30T06:46:21.381ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
82172025-04-30T06:46:21.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
82182025-04-30T06:46:21.385ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa80c835-3388-4510-b10a-e3b527c93620", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82192025-04-30T06:46:21.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
82202025-04-30T06:46:21.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b7aab16-6cc7-47c4-b625-3dc2ebd6c1e0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82212025-04-30T06:46:21.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
82222025-04-30T06:46:21.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "206a6a17-7050-4214-8cd4-ff11161f5e90", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82232025-04-30T06:46:21.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
82242025-04-30T06:46:21.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "273e1cda-5111-4621-bd2c-5f3dff300ea6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82252025-04-30T06:46:21.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
82262025-04-30T06:46:21.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a04da59-d498-46a5-951a-53e29fae93c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82272025-04-30T06:46:21.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
82282025-04-30T06:46:21.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dfaad29-34da-472e-aa8c-c55825f14852", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82292025-04-30T06:46:21.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
82302025-04-30T06:46:21.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b57c0bc-b95f-4405-b572-03eb6c4c597f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82312025-04-30T06:46:21.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
82322025-04-30T06:46:21.412ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
82332025-04-30T06:46:21.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3d7b5f9-a11e-4b5a-942c-e45922971314", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82342025-04-30T06:46:21.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
82352025-04-30T06:46:21.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3c491f4-b806-47f5-9603-cae5e1aecea0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82362025-04-30T06:46:21.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
82372025-04-30T06:46:21.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc8ffa8f-1f71-4195-9a22-c080e977724a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82382025-04-30T06:46:21.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
82392025-04-30T06:46:21.460ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee788d2-7bbd-47b3-99ee-271fb4904653", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82402025-04-30T06:46:21.460ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
82412025-04-30T06:46:21.462ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec5b6957-27e6-41f9-ab64-bb45ad6bdf34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82422025-04-30T06:46:21.462ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
82432025-04-30T06:46:21.462ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82442025-04-30T06:46:21.462ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82452025-04-30T06:46:21.462ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82462025-04-30T06:46:21.462ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
82472025-04-30T06:46:21.463ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82482025-04-30T06:46:21.466ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82492025-04-30T06:46:21.473ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82502025-04-30T06:46:21.482ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82512025-04-30T06:46:21.482ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
82522025-04-30T06:46:21.483ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82532025-04-30T06:46:21.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "defa5635-cc2f-402e-86bf-b8febe45e3d6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82542025-04-30T06:46:21.484ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
82552025-04-30T06:46:21.486ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82562025-04-30T06:46:21.486ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
82572025-04-30T06:46:21.487ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82582025-04-30T06:46:21.490ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a820407-8050-4bea-8cab-23b6726b4ac0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82592025-04-30T06:46:21.490ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
82602025-04-30T06:46:21.495ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51a40d1-6081-4031-8ba6-8d3dbb3c5b9e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82612025-04-30T06:46:21.495ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
82622025-04-30T06:46:21.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efbbe64a-7d50-48ee-9d9c-16b216778d87", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82632025-04-30T06:46:21.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
82642025-04-30T06:46:21.504ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b934b947-bbcd-49f0-9335-7e7f9dfff8b3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82652025-04-30T06:46:21.504ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
82662025-04-30T06:46:21.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ce39fa-fa08-4996-8d2d-ab3d3d87878c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82672025-04-30T06:46:21.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
82682025-04-30T06:46:21.541ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
82692025-04-30T06:46:21.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b61f5cdd-3c2e-482d-8a6f-4d0f804227c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82702025-04-30T06:46:21.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
82712025-04-30T06:46:21.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3b7b3d-d6ef-40d4-9fad-d8c8044dbbe1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82722025-04-30T06:46:21.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
82732025-04-30T06:46:21.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2717f1c-8641-408f-a43e-d52779101ee4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82742025-04-30T06:46:21.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
82752025-04-30T06:46:21.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
82762025-04-30T06:46:21.549ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82772025-04-30T06:46:21.549ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 502613b3-d2d4-4c2d-9a2b-f202e25ece97 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82782025-04-30T06:46:21.549ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 502613b3-d2d4-4c2d-9a2b-f202e25ece97 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82792025-04-30T06:46:21.549ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 502613b3-d2d4-4c2d-9a2b-f202e25ece97 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82802025-04-30T06:46:21.551ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 502613b3-d2d4-4c2d-9a2b-f202e25ece97 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82812025-04-30T06:46:21.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6184e225-7750-4eb7-b93b-a3c1ec0d6dc2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82822025-04-30T06:46:21.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
82832025-04-30T06:46:21.553ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7159423b-2115-426d-8165-942cee649886", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82842025-04-30T06:46:21.553ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
82852025-04-30T06:46:21.555ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6173 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 502613b3-d2d4-4c2d-9a2b-f202e25ece97 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
82862025-04-30T06:46:21.556ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "502613b3-d2d4-4c2d-9a2b-f202e25ece97", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82872025-04-30T06:46:21.556ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82882025-04-30T06:46:21.556ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82892025-04-30T06:46:21.556ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82902025-04-30T06:46:21.557ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82912025-04-30T06:46:21.557ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82922025-04-30T06:46:21.558ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc28210c-ee87-437e-b091-0b2f0abb811c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82932025-04-30T06:46:21.559ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
82942025-04-30T06:46:21.563ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6775 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
82952025-04-30T06:46:21.564ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fdac0f2-78c6-4e0e-adcd-b3fe08d06dae", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
82962025-04-30T06:46:21.564ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
82972025-04-30T06:46:21.564ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82982025-04-30T06:46:21.564ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
82992025-04-30T06:46:21.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cf8eabe-3049-4ae9-848a-99a6f6492f7d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83002025-04-30T06:46:21.567ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
83012025-04-30T06:46:21.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1da478d-d913-4cb4-a1ff-daea0f127e01", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83022025-04-30T06:46:21.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
83032025-04-30T06:46:21.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd27c04e-7355-41bc-883f-9bd294b2110a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83042025-04-30T06:46:21.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
83052025-04-30T06:46:21.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db8f9161-6b63-45e9-8468-e794fd007359", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83062025-04-30T06:46:21.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
83072025-04-30T06:46:21.579ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7823e63d-a46d-41c3-983a-955190e87110", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83082025-04-30T06:46:21.579ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
83092025-04-30T06:46:21.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff62234-145f-4b9d-a3de-bfc4825cb59d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83102025-04-30T06:46:21.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
83112025-04-30T06:46:21.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89bb0d2-8ce7-421f-a113-d7069063b329", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83122025-04-30T06:46:21.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
83132025-04-30T06:46:21.598ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83142025-04-30T06:46:21.598ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83152025-04-30T06:46:21.598ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83162025-04-30T06:46:21.598ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
83172025-04-30T06:46:21.599ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83182025-04-30T06:46:21.601ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9c042c4-0387-44a6-add4-b4a8a36426a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83192025-04-30T06:46:21.612ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
83202025-04-30T06:46:21.613ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83212025-04-30T06:46:21.620ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83222025-04-30T06:46:21.620ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83232025-04-30T06:46:21.620ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
83242025-04-30T06:46:21.622ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83252025-04-30T06:46:21.622ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22191155-5b0a-4209-b20e-c6b52c0fc399", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83262025-04-30T06:46:21.622ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
83272025-04-30T06:46:21.630ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83282025-04-30T06:46:21.630ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
83292025-04-30T06:46:21.631ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83302025-04-30T06:46:21.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f646df-f83b-4c64-a0b6-95f6200de5ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83312025-04-30T06:46:21.632ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
83322025-04-30T06:46:21.634ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52184839-e08e-4030-819d-752ccd7d48ef", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83332025-04-30T06:46:21.634ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
83342025-04-30T06:46:21.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83352025-04-30T06:46:21.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83362025-04-30T06:46:21.640ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83372025-04-30T06:46:21.641ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83382025-04-30T06:46:21.641ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83392025-04-30T06:46:21.641ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
83402025-04-30T06:46:21.642ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc77bcb1-9116-4320-97c8-191ae56ecc11", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83412025-04-30T06:46:21.643ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
83422025-04-30T06:46:21.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4df827c8-a2ae-4d67-8a06-75d632837b45", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83432025-04-30T06:46:21.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
83442025-04-30T06:46:21.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "771dd1d7-4727-47a8-a118-cf3a80c15bbe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83452025-04-30T06:46:21.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
83462025-04-30T06:46:21.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0199b154-0383-4951-9efc-dd527fe25869", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83472025-04-30T06:46:21.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
83482025-04-30T06:46:21.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f8cabb-a2f7-4dca-ab5d-8e6c90a8db0c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83492025-04-30T06:46:21.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
83502025-04-30T06:46:21.684ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
83512025-04-30T06:46:21.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "490ccf03-661b-4c78-890d-a44c310d138f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83522025-04-30T06:46:21.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
83532025-04-30T06:46:21.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27c2f0c9-9be2-4b7f-9759-7abe914d3713", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83542025-04-30T06:46:21.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
83552025-04-30T06:46:21.689ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26637d3c-c3c2-4813-baf5-f770c09acaba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83562025-04-30T06:46:21.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
83572025-04-30T06:46:21.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f68f719-4ab6-4426-9b0b-2a26d8bcb713", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83582025-04-30T06:46:21.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
83592025-04-30T06:46:21.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4be2753-2f32-4182-9457-dac2c30887d5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83602025-04-30T06:46:21.694ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83612025-04-30T06:46:21.694ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83622025-04-30T06:46:21.694ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83632025-04-30T06:46:21.694ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83642025-04-30T06:46:21.695ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83652025-04-30T06:46:21.699ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83662025-04-30T06:46:21.703ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83672025-04-30T06:46:21.703ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83682025-04-30T06:46:21.703ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83692025-04-30T06:46:21.703ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83702025-04-30T06:46:21.703ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83712025-04-30T06:46:21.704ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83722025-04-30T06:46:21.707ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83732025-04-30T06:46:21.712ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83742025-04-30T06:46:21.713ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83752025-04-30T06:46:21.713ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83762025-04-30T06:46:21.713ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83772025-04-30T06:46:21.777ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
83782025-04-30T06:46:21.777ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
83792025-04-30T06:46:21.778ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83802025-04-30T06:46:21.778ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83812025-04-30T06:46:21.778ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83822025-04-30T06:46:21.778ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
83832025-04-30T06:46:21.778ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83842025-04-30T06:46:21.779ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:47006 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83852025-04-30T06:46:21.779ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:47006 req_id = b25559e0-cf5f-4606-b328-37f3408f617f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83862025-04-30T06:46:21.780ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 599 local_addr = [::1]:54590 method = GET remote_addr = [::1]:47006 req_id = b25559e0-cf5f-4606-b328-37f3408f617f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83872025-04-30T06:46:21.780ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25559e0-cf5f-4606-b328-37f3408f617f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:21 GMT"} })
83882025-04-30T06:46:21.782ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83892025-04-30T06:46:21.782ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83902025-04-30T06:46:21.782ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
83912025-04-30T06:46:21.785ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83922025-04-30T06:46:21.785ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83932025-04-30T06:46:21.785ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83942025-04-30T06:46:21.785ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83952025-04-30T06:46:21.786ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83962025-04-30T06:46:21.786ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83972025-04-30T06:46:21.793ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83982025-04-30T06:46:21.793ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83992025-04-30T06:46:21.793ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
84002025-04-30T06:46:21.794ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84012025-04-30T06:46:21.797ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84022025-04-30T06:46:21.803ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84032025-04-30T06:46:21.804ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84042025-04-30T06:46:21.804ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84052025-04-30T06:46:21.811ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84062025-04-30T06:46:21.813ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84072025-04-30T06:46:21.813ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
84082025-04-30T06:46:21.814ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84092025-04-30T06:46:21.912ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84102025-04-30T06:46:21.912ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84112025-04-30T06:46:21.912ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 628.903489ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84122025-04-30T06:46:21.912ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84132025-04-30T06:46:21.912ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84142025-04-30T06:46:21.913ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84152025-04-30T06:46:21.913ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84162025-04-30T06:46:21.913ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84172025-04-30T06:46:21.913ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84182025-04-30T06:46:21.925ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84192025-04-30T06:46:21.958ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84202025-04-30T06:46:21.959ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84212025-04-30T06:46:21.960ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84222025-04-30T06:46:21.961ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84232025-04-30T06:46:21.972ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
84242025-04-30T06:46:21.972ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84252025-04-30T06:46:21.974ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
84262025-04-30T06:46:21.975ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 85 9e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60684 req_id = 7b1d0019-8485-4a44-9e89-b75475c6eea9
84272025-04-30T06:46:21.975ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60684 req_id = 7b1d0019-8485-4a44-9e89-b75475c6eea9
84282025-04-30T06:46:21.975ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
84292025-04-30T06:46:21.976ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84302025-04-30T06:46:21.976ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60684 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7b1d0019-8485-4a44-9e89-b75475c6eea9
84312025-04-30T06:46:21.977ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4446, tv_nsec: 776659207 } })
84322025-04-30T06:46:21.977ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
84332025-04-30T06:46:21.978ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 79 00 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e y...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51241 req_id = 6579b297-60a5-4a89-b3b9-564d123749c7
84342025-04-30T06:46:21.978ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30976,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51241 req_id = 6579b297-60a5-4a89-b3b9-564d123749c7
84352025-04-30T06:46:21.978ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84362025-04-30T06:46:21.979ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51241 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6579b297-60a5-4a89-b3b9-564d123749c7
84372025-04-30T06:46:22.089ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84382025-04-30T06:46:22.090ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84392025-04-30T06:46:22.090ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84402025-04-30T06:46:22.090ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84412025-04-30T06:46:22.090ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
84422025-04-30T06:46:22.091ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84432025-04-30T06:46:22.091ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84442025-04-30T06:46:22.091ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84452025-04-30T06:46:22.091ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
84462025-04-30T06:46:22.092ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
84472025-04-30T06:46:22.096ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 71 73 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c qs..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52909 req_id = 74767a8b-25bf-491b-8b8b-bf9641d984f8
84482025-04-30T06:46:22.096ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29043,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52909 req_id = 74767a8b-25bf-491b-8b8b-bf9641d984f8
84492025-04-30T06:46:22.097ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
84502025-04-30T06:46:22.097ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29043, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52909 req_id = 74767a8b-25bf-491b-8b8b-bf9641d984f8
84512025-04-30T06:46:22.097ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84522025-04-30T06:46:22.107ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84532025-04-30T06:46:22.107ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84542025-04-30T06:46:22.108ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
84552025-04-30T06:46:22.108ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84562025-04-30T06:46:22.109ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
84572025-04-30T06:46:22.109ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
84582025-04-30T06:46:22.117ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
84592025-04-30T06:46:22.117ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
84602025-04-30T06:46:22.117ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
84612025-04-30T06:46:22.121ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84622025-04-30T06:46:22.121ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
84632025-04-30T06:46:22.122ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84642025-04-30T06:46:22.122ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
84652025-04-30T06:46:22.123ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
84662025-04-30T06:46:22.129ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "387d503b-e9bb-4daf-a0da-0562d2e789e7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84672025-04-30T06:46:22.130ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
84682025-04-30T06:46:22.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f3aede-cbf5-48ee-aa49-581e7a404d2c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84692025-04-30T06:46:22.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
84702025-04-30T06:46:22.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0818cd35-4a13-40d4-b25d-544f7cefdaed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84712025-04-30T06:46:22.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
84722025-04-30T06:46:22.153ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9298933a-06ec-449f-b80d-dd81537bc5e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84732025-04-30T06:46:22.155ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
84742025-04-30T06:46:22.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f46bf1e-65a4-41d2-b67b-e2d0cd6c358a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84752025-04-30T06:46:22.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
84762025-04-30T06:46:22.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e67bb2-4fc2-4646-aecf-4fb4d275a4a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84772025-04-30T06:46:22.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
84782025-04-30T06:46:22.167ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
84792025-04-30T06:46:22.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d8d98d-95d2-456d-a3f3-d5a04417c29d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84802025-04-30T06:46:22.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
84812025-04-30T06:46:22.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2dad5d4-9e21-4ce9-8647-993bfe7a3dbb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84822025-04-30T06:46:22.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
84832025-04-30T06:46:22.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21357522-ece4-4dd6-8fab-bb429f53ad5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84842025-04-30T06:46:22.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
84852025-04-30T06:46:22.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a6b3234-30e3-413b-95f0-59440be0acf9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84862025-04-30T06:46:22.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
84872025-04-30T06:46:22.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0a5d118-cb80-408f-9002-658d62bb28ff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84882025-04-30T06:46:22.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
84892025-04-30T06:46:22.192ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
84902025-04-30T06:46:22.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d3dba8-5dc0-41cc-9c6c-cf448e9aa7b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84912025-04-30T06:46:22.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
84922025-04-30T06:46:22.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "480fd482-e791-4601-8077-4f7311ad77fa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84932025-04-30T06:46:22.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
84942025-04-30T06:46:22.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a7dbae2-96c9-4e42-bec5-c4fcc7db1483", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84952025-04-30T06:46:22.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
84962025-04-30T06:46:22.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "166bbeaf-16a7-4532-95e1-5fbe15c81b8a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
84972025-04-30T06:46:22.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
84982025-04-30T06:46:22.219ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84992025-04-30T06:46:22.219ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85002025-04-30T06:46:22.219ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85012025-04-30T06:46:22.219ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85022025-04-30T06:46:22.220ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85032025-04-30T06:46:22.232ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85042025-04-30T06:46:22.238ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85052025-04-30T06:46:22.238ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85062025-04-30T06:46:22.238ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85072025-04-30T06:46:22.239ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85082025-04-30T06:46:22.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00116652-9cc8-4b1a-b5ec-d620616e39d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85092025-04-30T06:46:22.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
85102025-04-30T06:46:22.244ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85112025-04-30T06:46:22.244ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85122025-04-30T06:46:22.244ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85132025-04-30T06:46:22.254ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "324af3cc-4e2a-44a3-9801-cec863044382", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85142025-04-30T06:46:22.255ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
85152025-04-30T06:46:22.258ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ae71be-ca28-4c97-9b73-f7592297167b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85162025-04-30T06:46:22.262ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
85172025-04-30T06:46:22.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e0cf38f-b7cc-40f1-aab9-63e3610e63a4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85182025-04-30T06:46:22.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
85192025-04-30T06:46:22.269ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e2f810-922b-4ac9-8a8d-9a1e6eea3cc4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85202025-04-30T06:46:22.269ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
85212025-04-30T06:46:22.272ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d5350de-c116-40dc-8f01-f11994ca851c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85222025-04-30T06:46:22.272ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
85232025-04-30T06:46:22.278ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "450959d8-6cff-4663-aba9-2a35793a0860", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85242025-04-30T06:46:22.278ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
85252025-04-30T06:46:22.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dba0a2c-8e40-4c8d-b72f-98aec439a104", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85262025-04-30T06:46:22.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
85272025-04-30T06:46:22.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "543b825f-80ba-4086-b060-29b122ea47a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85282025-04-30T06:46:22.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
85292025-04-30T06:46:22.286ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
85302025-04-30T06:46:22.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d0e0bf-29a8-44f1-9b46-af54842618f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85312025-04-30T06:46:22.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
85322025-04-30T06:46:22.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5865e0df-a7bd-468d-9bdd-3878f8585286", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85332025-04-30T06:46:22.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
85342025-04-30T06:46:22.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3577ad9a-9678-47e8-9cc4-e570b57ac66b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85352025-04-30T06:46:22.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
85362025-04-30T06:46:22.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd3d185a-c4b6-45e2-b6b2-7f80f10084bb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85372025-04-30T06:46:22.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
85382025-04-30T06:46:22.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fefbd67-542f-4f54-86ae-be4364d4f1b2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85392025-04-30T06:46:22.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
85402025-04-30T06:46:22.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41d9a52b-f84f-4f93-bf83-e82dede21686", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85412025-04-30T06:46:22.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
85422025-04-30T06:46:22.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd62301e-0381-4d71-96f4-7daf4e4d1aa8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85432025-04-30T06:46:22.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
85442025-04-30T06:46:22.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb27651-a18c-4042-a353-86aa0058d9be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85452025-04-30T06:46:22.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
85462025-04-30T06:46:22.306ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36cfb4d-1060-40e7-9830-2aa98d4f8f32", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85472025-04-30T06:46:22.306ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
85482025-04-30T06:46:22.308ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28d07f91-4c73-4350-8544-80b20b652265", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85492025-04-30T06:46:22.308ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
85502025-04-30T06:46:22.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d09b2aa2-22fe-4bd2-ba4f-63b243cfe79c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85512025-04-30T06:46:22.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
85522025-04-30T06:46:22.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b15ab46a-6f45-42a6-9968-a30eb8f16613", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85532025-04-30T06:46:22.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
85542025-04-30T06:46:22.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cd786eb-34a4-465a-9f91-16ca5214faef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85552025-04-30T06:46:22.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
85562025-04-30T06:46:22.316ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c53dbc12-4939-4d0b-a0b8-731a86ca15f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85572025-04-30T06:46:22.316ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
85582025-04-30T06:46:22.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3ac6384-57d5-4848-bc0e-ca74cdae209e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85592025-04-30T06:46:22.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
85602025-04-30T06:46:22.338ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85612025-04-30T06:46:22.338ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85622025-04-30T06:46:22.338ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85632025-04-30T06:46:22.338ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85642025-04-30T06:46:22.338ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85652025-04-30T06:46:22.341ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85662025-04-30T06:46:22.352ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85672025-04-30T06:46:22.352ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85682025-04-30T06:46:22.352ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85692025-04-30T06:46:22.353ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85702025-04-30T06:46:22.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ea709a4-0ede-4a2c-a7c8-e9ff7bba08ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85712025-04-30T06:46:22.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
85722025-04-30T06:46:22.356ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85732025-04-30T06:46:22.356ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
85742025-04-30T06:46:22.356ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85752025-04-30T06:46:22.369ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1783d673-6b40-467a-adfb-840c4cd85bed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85762025-04-30T06:46:22.369ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
85772025-04-30T06:46:22.371ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcef994f-fc29-43b9-aedf-fcc4b020e158", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85782025-04-30T06:46:22.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
85792025-04-30T06:46:22.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bd5bb7-8bef-4c4b-b31e-0c93495bc016", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85802025-04-30T06:46:22.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
85812025-04-30T06:46:22.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70bb7c8b-a6b7-4a53-9626-c3caa9a150d5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85822025-04-30T06:46:22.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
85832025-04-30T06:46:22.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29f21917-f19e-419a-80fb-7772a48f4677", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
85842025-04-30T06:46:22.379ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85852025-04-30T06:46:22.379ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85862025-04-30T06:46:22.379ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85872025-04-30T06:46:22.379ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85882025-04-30T06:46:22.380ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85892025-04-30T06:46:22.383ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85902025-04-30T06:46:22.389ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85912025-04-30T06:46:22.389ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85922025-04-30T06:46:22.389ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85932025-04-30T06:46:22.389ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85942025-04-30T06:46:22.389ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85952025-04-30T06:46:22.390ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85962025-04-30T06:46:22.397ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85972025-04-30T06:46:22.403ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85982025-04-30T06:46:22.404ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85992025-04-30T06:46:22.404ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86002025-04-30T06:46:22.405ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86012025-04-30T06:46:22.506ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
86022025-04-30T06:46:22.506ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
86032025-04-30T06:46:22.509ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:42033 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86042025-04-30T06:46:22.510ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:42033 req_id = 20a64a19-2505-4456-b92c-cd51193198fe sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86052025-04-30T06:46:22.511ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 657 local_addr = [::1]:54590 method = GET remote_addr = [::1]:42033 req_id = 20a64a19-2505-4456-b92c-cd51193198fe response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86062025-04-30T06:46:22.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a64a19-2505-4456-b92c-cd51193198fe", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86072025-04-30T06:46:22.516ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86082025-04-30T06:46:22.516ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86092025-04-30T06:46:22.516ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
86102025-04-30T06:46:22.519ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86112025-04-30T06:46:22.519ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86122025-04-30T06:46:22.519ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86132025-04-30T06:46:22.519ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86142025-04-30T06:46:22.520ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86152025-04-30T06:46:22.531ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86162025-04-30T06:46:22.536ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86172025-04-30T06:46:22.538ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86182025-04-30T06:46:22.538ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86192025-04-30T06:46:22.538ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86202025-04-30T06:46:22.539ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
86212025-04-30T06:46:22.593ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86222025-04-30T06:46:22.595ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86232025-04-30T06:46:22.595ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86242025-04-30T06:46:22.595ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
86252025-04-30T06:46:22.597ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86262025-04-30T06:46:22.615ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86272025-04-30T06:46:22.621ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86282025-04-30T06:46:22.621ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86292025-04-30T06:46:22.621ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
86302025-04-30T06:46:22.622ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86312025-04-30T06:46:22.637ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86322025-04-30T06:46:22.637ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
86332025-04-30T06:46:22.638ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86342025-04-30T06:46:22.639ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86352025-04-30T06:46:22.639ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86362025-04-30T06:46:22.639ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86372025-04-30T06:46:22.640ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86382025-04-30T06:46:22.640ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 727.549996ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86392025-04-30T06:46:22.640ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86402025-04-30T06:46:22.640ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86412025-04-30T06:46:22.640ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86422025-04-30T06:46:22.640ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86432025-04-30T06:46:22.640ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86442025-04-30T06:46:22.641ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86452025-04-30T06:46:22.648ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86462025-04-30T06:46:22.665ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86472025-04-30T06:46:22.665ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86482025-04-30T06:46:22.665ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86492025-04-30T06:46:22.665ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86502025-04-30T06:46:22.669ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86512025-04-30T06:46:22.669ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86522025-04-30T06:46:22.669ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86532025-04-30T06:46:22.671ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86542025-04-30T06:46:22.676ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
86552025-04-30T06:46:22.677ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86562025-04-30T06:46:22.678ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4b 6b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Kk..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49303 req_id = 594ba97d-07f9-4ef3-884a-c05b3de23082
86572025-04-30T06:46:22.678ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19307,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49303 req_id = 594ba97d-07f9-4ef3-884a-c05b3de23082
86582025-04-30T06:46:22.679ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
86592025-04-30T06:46:22.679ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86602025-04-30T06:46:22.679ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49303 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 594ba97d-07f9-4ef3-884a-c05b3de23082
86612025-04-30T06:46:22.680ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4447, tv_nsec: 479136327 } })
86622025-04-30T06:46:22.680ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
86632025-04-30T06:46:22.680ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 03 90 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55544 req_id = c415e587-6d2c-446a-8225-511e40d5b5f8
86642025-04-30T06:46:22.680ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55544 req_id = c415e587-6d2c-446a-8225-511e40d5b5f8
86652025-04-30T06:46:22.681ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86662025-04-30T06:46:22.681ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55544 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c415e587-6d2c-446a-8225-511e40d5b5f8
86672025-04-30T06:46:22.815ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86682025-04-30T06:46:22.816ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c5 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64136 req_id = 7d9998c4-9f30-42d6-9c13-b3d80ba22c14
86692025-04-30T06:46:22.816ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50450,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64136 req_id = 7d9998c4-9f30-42d6-9c13-b3d80ba22c14
86702025-04-30T06:46:22.817ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
86712025-04-30T06:46:22.817ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50450, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64136 req_id = 7d9998c4-9f30-42d6-9c13-b3d80ba22c14
86722025-04-30T06:46:22.830ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
86732025-04-30T06:46:22.830ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
86742025-04-30T06:46:22.830ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
86752025-04-30T06:46:22.842ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b7e451-91a9-4532-9236-96838ac51b43", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86762025-04-30T06:46:22.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
86772025-04-30T06:46:22.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97f83138-508a-49d8-bda7-2b264bcd9bfc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86782025-04-30T06:46:22.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
86792025-04-30T06:46:22.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74e2c77-dc73-4d40-80c0-bdcf5ee9b57e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86802025-04-30T06:46:22.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
86812025-04-30T06:46:22.860ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
86822025-04-30T06:46:22.874ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f71604fe-2d8f-48bd-bcb7-bc09448601e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86832025-04-30T06:46:22.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
86842025-04-30T06:46:22.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4fa01d0-accf-4d58-bad8-c2181242df03", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86852025-04-30T06:46:22.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
86862025-04-30T06:46:22.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed0df48-728b-4af8-b38e-3ec6c700ece0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86872025-04-30T06:46:22.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
86882025-04-30T06:46:22.919ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86892025-04-30T06:46:22.919ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86902025-04-30T06:46:22.920ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86912025-04-30T06:46:22.921ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
86922025-04-30T06:46:22.922ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86932025-04-30T06:46:22.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb044f9-22fa-4836-8bfd-99fb7b0a9d23", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
86942025-04-30T06:46:22.923ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
86952025-04-30T06:46:22.925ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86962025-04-30T06:46:22.934ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86972025-04-30T06:46:22.939ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86982025-04-30T06:46:22.939ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
86992025-04-30T06:46:22.940ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87002025-04-30T06:46:22.947ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87012025-04-30T06:46:22.953ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
87022025-04-30T06:46:22.958ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87032025-04-30T06:46:22.963ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68425e1-134e-46d3-91d5-c24aa3845838", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87042025-04-30T06:46:22.967ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
87052025-04-30T06:46:22.969ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9b23e72-98f1-487c-8e8d-c02f77e30f9b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87062025-04-30T06:46:22.969ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
87072025-04-30T06:46:22.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cebc4945-2959-40e3-bfd1-b37d00f48714", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87082025-04-30T06:46:22.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
87092025-04-30T06:46:22.988ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87102025-04-30T06:46:22.990ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87112025-04-30T06:46:22.991ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
87122025-04-30T06:46:22.992ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
87132025-04-30T06:46:23.015ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
87142025-04-30T06:46:23.015ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
87152025-04-30T06:46:23.017ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d6fde54-0740-4fc0-a158-68bc66708450", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87162025-04-30T06:46:23.017ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
87172025-04-30T06:46:23.024ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
87182025-04-30T06:46:23.024ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
87192025-04-30T06:46:23.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ba92317-9416-4433-886d-f6a7c0fcdfb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87202025-04-30T06:46:23.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
87212025-04-30T06:46:23.044ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8de5685-a7c7-470a-8ff7-5c93daf57959", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87222025-04-30T06:46:23.044ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
87232025-04-30T06:46:23.072ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21bbaa87-db18-40d9-8780-54e533ba8404", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87242025-04-30T06:46:23.072ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
87252025-04-30T06:46:23.073ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
87262025-04-30T06:46:23.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c0069cb-5f8a-459b-92aa-bdaf5327abbf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87272025-04-30T06:46:23.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
87282025-04-30T06:46:23.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4553804-0790-4b1f-be9e-fbd3fa791e4a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87292025-04-30T06:46:23.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
87302025-04-30T06:46:23.092ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
87312025-04-30T06:46:23.093ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32844be0-d921-4f14-ade2-92ba6701f998", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87322025-04-30T06:46:23.093ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
87332025-04-30T06:46:23.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a5ae0d8-36c9-4c79-b24c-e1428c6e67f6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87342025-04-30T06:46:23.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
87352025-04-30T06:46:23.098ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d15333f-adb1-4a32-8247-39c8ce7cfb3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87362025-04-30T06:46:23.098ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
87372025-04-30T06:46:23.104ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d60ba3c-36a8-4943-ab77-c449a2a884b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87382025-04-30T06:46:23.105ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
87392025-04-30T06:46:23.106ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb9a6fe-da81-4ab9-803a-63797817de63", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87402025-04-30T06:46:23.106ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
87412025-04-30T06:46:23.121ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5247d3-117e-4a02-8ed3-4c08ff614b06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:22 GMT"} })
87422025-04-30T06:46:23.121ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
87432025-04-30T06:46:23.125ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "655b6d58-89f5-4d27-af0d-520bc2f1c76a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87442025-04-30T06:46:23.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
87452025-04-30T06:46:23.126ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87462025-04-30T06:46:23.126ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87472025-04-30T06:46:23.126ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87482025-04-30T06:46:23.126ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
87492025-04-30T06:46:23.127ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87502025-04-30T06:46:23.129ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f10822f7-2659-4f49-a10e-11ae6cd87e17", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87512025-04-30T06:46:23.129ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
87522025-04-30T06:46:23.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b77da183-ba2c-4ee7-b301-84038009e9d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87532025-04-30T06:46:23.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
87542025-04-30T06:46:23.132ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87552025-04-30T06:46:23.138ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87562025-04-30T06:46:23.138ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87572025-04-30T06:46:23.138ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
87582025-04-30T06:46:23.138ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87592025-04-30T06:46:23.168ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87602025-04-30T06:46:23.168ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
87612025-04-30T06:46:23.174ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87622025-04-30T06:46:23.204ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c912bd2c-38e3-4f67-8ebb-0da2bceed295", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87632025-04-30T06:46:23.205ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
87642025-04-30T06:46:23.207ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a59102-a8cf-4f6a-9c4b-790bc878a2e7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87652025-04-30T06:46:23.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
87662025-04-30T06:46:23.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ba1c097-8fb0-45f9-bf74-83fbace00b14", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87672025-04-30T06:46:23.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
87682025-04-30T06:46:23.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6a386b-7458-4f57-a645-5eee2f2a3cad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87692025-04-30T06:46:23.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
87702025-04-30T06:46:23.221ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a51dd6dc-8c14-495b-b2f1-9a7ab5f1ae0c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87712025-04-30T06:46:23.222ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
87722025-04-30T06:46:23.224ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfee9b83-aab2-40d3-8c88-2263f20d4d40", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87732025-04-30T06:46:23.253ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
87742025-04-30T06:46:23.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6fed0f7-d99b-4ea7-9742-487edfea6a3a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87752025-04-30T06:46:23.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
87762025-04-30T06:46:23.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68f13bf-a285-47cd-831d-e2155039ed9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87772025-04-30T06:46:23.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
87782025-04-30T06:46:23.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad05a9b5-bbb3-45c3-b009-67b7b884679a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87792025-04-30T06:46:23.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
87802025-04-30T06:46:23.270ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6765c442-807b-4447-b5e0-c8068359ddfa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87812025-04-30T06:46:23.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
87822025-04-30T06:46:23.276ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
87832025-04-30T06:46:23.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "574ca41d-a036-452f-a762-d08a41601d4d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87842025-04-30T06:46:23.280ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
87852025-04-30T06:46:23.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12065cce-3014-4d12-b090-e8d7c59d2eed", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87862025-04-30T06:46:23.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
87872025-04-30T06:46:23.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2648449c-a9dc-411c-97f8-42cd9d5db79e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87882025-04-30T06:46:23.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
87892025-04-30T06:46:23.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c00a914e-16a1-487f-b80e-52aef4c63a79", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87902025-04-30T06:46:23.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
87912025-04-30T06:46:23.295ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cbf5d2f-9bc3-4895-85b1-3d1f9b722524", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87922025-04-30T06:46:23.295ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
87932025-04-30T06:46:23.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8eaa384-5a4b-40ab-9ef0-5b0b204d929d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87942025-04-30T06:46:23.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
87952025-04-30T06:46:23.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f4f8c71-e1ee-450b-840b-7a1b46435b01", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87962025-04-30T06:46:23.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
87972025-04-30T06:46:23.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3169de52-bfd7-4dd0-bc7a-37cdd416f349", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
87982025-04-30T06:46:23.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
87992025-04-30T06:46:23.300ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "738afe12-ffd4-45c2-9592-41eacc24f69a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
88002025-04-30T06:46:23.300ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
88012025-04-30T06:46:23.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c03ce0-e50a-4652-9d95-c859b7f4145b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
88022025-04-30T06:46:23.301ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88032025-04-30T06:46:23.301ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88042025-04-30T06:46:23.301ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88052025-04-30T06:46:23.301ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88062025-04-30T06:46:23.302ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88072025-04-30T06:46:23.354ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88082025-04-30T06:46:23.354ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88092025-04-30T06:46:23.354ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88102025-04-30T06:46:23.354ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
88112025-04-30T06:46:23.355ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88122025-04-30T06:46:23.355ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88132025-04-30T06:46:23.359ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88142025-04-30T06:46:23.359ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88152025-04-30T06:46:23.359ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88162025-04-30T06:46:23.359ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88172025-04-30T06:46:23.359ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88182025-04-30T06:46:23.359ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88192025-04-30T06:46:23.362ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88202025-04-30T06:46:23.368ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88212025-04-30T06:46:23.368ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88222025-04-30T06:46:23.368ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
88232025-04-30T06:46:23.369ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88242025-04-30T06:46:23.370ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88252025-04-30T06:46:23.375ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88262025-04-30T06:46:23.375ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88272025-04-30T06:46:23.375ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88282025-04-30T06:46:23.375ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88292025-04-30T06:46:23.401ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88302025-04-30T06:46:23.401ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
88312025-04-30T06:46:23.406ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88322025-04-30T06:46:23.468ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
88332025-04-30T06:46:23.468ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
88342025-04-30T06:46:23.469ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:58628 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88352025-04-30T06:46:23.472ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:58628 req_id = 0a3f04f3-5547-4f8e-a16e-a88443208a37 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88362025-04-30T06:46:23.473ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 670 local_addr = [::1]:54590 method = GET remote_addr = [::1]:58628 req_id = 0a3f04f3-5547-4f8e-a16e-a88443208a37 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88372025-04-30T06:46:23.479ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a3f04f3-5547-4f8e-a16e-a88443208a37", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
88382025-04-30T06:46:23.480ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88392025-04-30T06:46:23.480ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88402025-04-30T06:46:23.481ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
88412025-04-30T06:46:23.482ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88422025-04-30T06:46:23.482ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88432025-04-30T06:46:23.482ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88442025-04-30T06:46:23.482ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88452025-04-30T06:46:23.483ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88462025-04-30T06:46:23.489ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88472025-04-30T06:46:23.494ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88482025-04-30T06:46:23.494ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88492025-04-30T06:46:23.495ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88502025-04-30T06:46:23.495ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88512025-04-30T06:46:23.565ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
88522025-04-30T06:46:23.581ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88532025-04-30T06:46:23.583ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 747edcfd-d697-42c2-987b-ef3398bbcf7a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88542025-04-30T06:46:23.583ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 747edcfd-d697-42c2-987b-ef3398bbcf7a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88552025-04-30T06:46:23.583ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 747edcfd-d697-42c2-987b-ef3398bbcf7a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88562025-04-30T06:46:23.584ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 747edcfd-d697-42c2-987b-ef3398bbcf7a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88572025-04-30T06:46:23.609ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 26393 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 747edcfd-d697-42c2-987b-ef3398bbcf7a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88582025-04-30T06:46:23.610ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "747edcfd-d697-42c2-987b-ef3398bbcf7a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
88592025-04-30T06:46:23.610ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88602025-04-30T06:46:23.610ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6c6b429b-13cf-4512-ab5b-cd6a8b785245 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88612025-04-30T06:46:23.610ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6c6b429b-13cf-4512-ab5b-cd6a8b785245 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88622025-04-30T06:46:23.611ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6c6b429b-13cf-4512-ab5b-cd6a8b785245 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88632025-04-30T06:46:23.611ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6c6b429b-13cf-4512-ab5b-cd6a8b785245 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88642025-04-30T06:46:23.625ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 14697 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6c6b429b-13cf-4512-ab5b-cd6a8b785245 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88652025-04-30T06:46:23.627ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
88662025-04-30T06:46:23.627ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c6b429b-13cf-4512-ab5b-cd6a8b785245", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
88672025-04-30T06:46:23.627ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
88682025-04-30T06:46:23.627ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88692025-04-30T06:46:23.627ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
88702025-04-30T06:46:23.649ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88712025-04-30T06:46:23.649ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88722025-04-30T06:46:23.649ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.009362228s iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88732025-04-30T06:46:23.649ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88742025-04-30T06:46:23.650ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88752025-04-30T06:46:23.650ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88762025-04-30T06:46:23.650ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88772025-04-30T06:46:23.650ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88782025-04-30T06:46:23.651ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88792025-04-30T06:46:23.651ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88802025-04-30T06:46:23.651ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88812025-04-30T06:46:23.653ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88822025-04-30T06:46:23.653ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88832025-04-30T06:46:23.654ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88842025-04-30T06:46:23.654ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88852025-04-30T06:46:23.661ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88862025-04-30T06:46:23.668ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88872025-04-30T06:46:23.668ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88882025-04-30T06:46:23.668ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88892025-04-30T06:46:23.668ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88902025-04-30T06:46:23.673ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
88912025-04-30T06:46:23.678ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88922025-04-30T06:46:23.679ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88932025-04-30T06:46:23.680ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88942025-04-30T06:46:23.680ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88952025-04-30T06:46:23.680ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
88962025-04-30T06:46:23.685ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88972025-04-30T06:46:23.685ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 74 ee 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 t............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55689 req_id = 6272f78c-b74e-4ce1-aa0e-197355ca411f
88982025-04-30T06:46:23.686ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29934,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55689 req_id = 6272f78c-b74e-4ce1-aa0e-197355ca411f
88992025-04-30T06:46:23.686ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
89002025-04-30T06:46:23.686ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89012025-04-30T06:46:23.687ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55689 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6272f78c-b74e-4ce1-aa0e-197355ca411f
89022025-04-30T06:46:23.688ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4448, tv_nsec: 487325447 } })
89032025-04-30T06:46:23.688ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
89042025-04-30T06:46:23.689ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89052025-04-30T06:46:23.698ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89062025-04-30T06:46:23.698ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89072025-04-30T06:46:23.699ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
89082025-04-30T06:46:23.701ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89092025-04-30T06:46:23.701ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4a ae 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e J...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60012 req_id = 9be1aa1c-2bdc-45d4-a93e-14c0bcb1575f
89102025-04-30T06:46:23.701ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19118,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60012 req_id = 9be1aa1c-2bdc-45d4-a93e-14c0bcb1575f
89112025-04-30T06:46:23.701ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89122025-04-30T06:46:23.702ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60012 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9be1aa1c-2bdc-45d4-a93e-14c0bcb1575f
89132025-04-30T06:46:23.807ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
89142025-04-30T06:46:23.808ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6e 4c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c nL..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55483 req_id = b2ec61f7-6076-4f58-b3ce-2e63db586148
89152025-04-30T06:46:23.809ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28236,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55483 req_id = b2ec61f7-6076-4f58-b3ce-2e63db586148
89162025-04-30T06:46:23.809ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
89172025-04-30T06:46:23.813ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55483 req_id = b2ec61f7-6076-4f58-b3ce-2e63db586148
89182025-04-30T06:46:23.813ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89192025-04-30T06:46:23.813ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
89202025-04-30T06:46:23.815ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89212025-04-30T06:46:23.815ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
89222025-04-30T06:46:23.815ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
89232025-04-30T06:46:23.815ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
89242025-04-30T06:46:23.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "768de91c-5709-4397-8849-64a332d2123a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89252025-04-30T06:46:23.840ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
89262025-04-30T06:46:23.844ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60a60135-3875-4def-a844-32bbd86575a7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89272025-04-30T06:46:23.844ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
89282025-04-30T06:46:23.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e3f9368-3b5f-4d6c-91a5-f3c3300db1ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89292025-04-30T06:46:23.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
89302025-04-30T06:46:23.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c958e14-1071-450a-8fae-ce14d69fd630", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89312025-04-30T06:46:23.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
89322025-04-30T06:46:23.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "879c8cc1-1f43-466a-822a-5c7fdf855df7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89332025-04-30T06:46:23.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
89342025-04-30T06:46:23.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcbbded4-bf7d-4035-b1c8-2ca01e77e4cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89352025-04-30T06:46:23.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
89362025-04-30T06:46:23.878ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
89372025-04-30T06:46:23.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c00feb0-d48f-4880-973c-3d827ed4d894", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89382025-04-30T06:46:23.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
89392025-04-30T06:46:23.890ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16c2cc17-aaae-4b20-86bd-c50c21acdd6f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89402025-04-30T06:46:23.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
89412025-04-30T06:46:23.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55d6fe3e-0dc1-4bdc-ac22-49d46d8430be", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89422025-04-30T06:46:23.893ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
89432025-04-30T06:46:23.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d3f258a-32f0-42d1-98f6-c5b54841e2c8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89442025-04-30T06:46:23.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
89452025-04-30T06:46:23.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39759a52-0139-4919-adea-d24fec2d2227", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89462025-04-30T06:46:23.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
89472025-04-30T06:46:23.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07fe9e01-b87a-481f-a160-066f2e8f7beb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89482025-04-30T06:46:23.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
89492025-04-30T06:46:23.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9301868-56aa-4cec-b288-2473f98e4274", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89502025-04-30T06:46:23.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
89512025-04-30T06:46:23.926ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "608f0a5b-09b8-4638-81d9-0def8aede088", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89522025-04-30T06:46:23.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
89532025-04-30T06:46:23.929ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89542025-04-30T06:46:23.929ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89552025-04-30T06:46:23.930ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89562025-04-30T06:46:23.930ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
89572025-04-30T06:46:23.931ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89582025-04-30T06:46:23.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "613549ee-9531-41be-8d92-3021ac04dc2e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89592025-04-30T06:46:23.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
89602025-04-30T06:46:23.969ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89612025-04-30T06:46:23.999ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89622025-04-30T06:46:23.999ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89632025-04-30T06:46:23.999ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
89642025-04-30T06:46:24.000ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89652025-04-30T06:46:24.002ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89662025-04-30T06:46:24.002ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89672025-04-30T06:46:24.002ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
89682025-04-30T06:46:24.004ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
89692025-04-30T06:46:24.007ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3028e9f2-ff15-4948-9d6a-e3f1644b5882", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89702025-04-30T06:46:24.007ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
89712025-04-30T06:46:24.007ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89722025-04-30T06:46:24.007ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
89732025-04-30T06:46:24.008ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89742025-04-30T06:46:24.009ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
89752025-04-30T06:46:24.009ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
89762025-04-30T06:46:24.018ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
89772025-04-30T06:46:24.018ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89782025-04-30T06:46:24.019ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ff2206c-a36e-4bea-9c45-cdddc5a5adaf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89792025-04-30T06:46:24.019ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
89802025-04-30T06:46:24.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2425b57e-23e1-424f-a567-57690e9f9f0a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89812025-04-30T06:46:24.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
89822025-04-30T06:46:24.026ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fcb635e-7287-4213-99ba-f3204cf6b0b9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89832025-04-30T06:46:24.026ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
89842025-04-30T06:46:24.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e68f6ab-a3d2-42ae-a292-65e451fc8435", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89852025-04-30T06:46:24.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
89862025-04-30T06:46:24.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41ffcc6c-149c-4f12-a155-97a0b33e58af", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89872025-04-30T06:46:24.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
89882025-04-30T06:46:24.031ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76a0f955-81cf-4bb2-8e3c-51f4576021a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89892025-04-30T06:46:24.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
89902025-04-30T06:46:24.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "965448e4-c250-43b6-9c76-57aa98d86b6a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89912025-04-30T06:46:24.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
89922025-04-30T06:46:24.036ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ab189a-0658-4a63-a0d4-15896fb812da", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89932025-04-30T06:46:24.036ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
89942025-04-30T06:46:24.040ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "662e05a7-1383-4c38-94ff-acd2d7519e0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89952025-04-30T06:46:24.040ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
89962025-04-30T06:46:24.061ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
89972025-04-30T06:46:24.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a179d74a-aace-4e67-80a8-bbe05e598fc8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
89982025-04-30T06:46:24.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
89992025-04-30T06:46:24.066ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5e52c0-b83e-428a-bebd-d1dabf3856fd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90002025-04-30T06:46:24.068ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
90012025-04-30T06:46:24.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f5f78b-547b-42cd-8f5c-a3ee719495a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90022025-04-30T06:46:24.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
90032025-04-30T06:46:24.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ba5683-a4bd-442f-bae3-d5aefa617ebe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90042025-04-30T06:46:24.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
90052025-04-30T06:46:24.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04430b05-d75e-4856-b9c7-9cf3e50e3d4e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90062025-04-30T06:46:24.077ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
90072025-04-30T06:46:24.078ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "703a813f-d884-4fb4-b9a7-b68dae7a8f2f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90082025-04-30T06:46:24.078ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
90092025-04-30T06:46:24.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "559a53c4-e37b-4ba3-8f36-4611b4ae2d4f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90102025-04-30T06:46:24.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
90112025-04-30T06:46:24.080ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f087b105-076c-4013-85c2-fe2f18407814", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90122025-04-30T06:46:24.080ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
90132025-04-30T06:46:24.081ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e7713e8-6d4f-4b8b-8866-8cff4e5ce12c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90142025-04-30T06:46:24.081ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
90152025-04-30T06:46:24.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04a95eaf-f1e2-4815-a768-307874c2db83", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90162025-04-30T06:46:24.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
90172025-04-30T06:46:24.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "752ea2f9-9da2-47f5-8641-4a78cbae9875", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90182025-04-30T06:46:24.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
90192025-04-30T06:46:24.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c070c053-49df-4467-97db-a363c78ba659", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90202025-04-30T06:46:24.087ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
90212025-04-30T06:46:24.090ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "309f31b2-b21f-4a70-b2bb-20175b632971", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90222025-04-30T06:46:24.090ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
90232025-04-30T06:46:24.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5704116b-0dc0-4436-ad9c-33359f89e147", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90242025-04-30T06:46:24.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
90252025-04-30T06:46:24.093ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "006e3046-ddc8-405f-b5bb-6c5bb57b6a54", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90262025-04-30T06:46:24.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
90272025-04-30T06:46:24.096ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "358f4b2c-7e79-4fc4-8f47-305f03108a50", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90282025-04-30T06:46:24.096ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
90292025-04-30T06:46:24.097ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd6bf26-829b-48e4-a621-4beaa200de1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90302025-04-30T06:46:24.097ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
90312025-04-30T06:46:24.098ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f979f84a-31fc-4cdb-8d3c-4230780f155b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90322025-04-30T06:46:24.098ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
90332025-04-30T06:46:24.099ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eab01eb6-5645-4648-bad6-a4b8abbbca99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90342025-04-30T06:46:24.099ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
90352025-04-30T06:46:24.100ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81cd361a-fe43-4b89-b17d-fd29ee3a7638", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:23 GMT"} })
90362025-04-30T06:46:24.100ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90372025-04-30T06:46:24.100ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90382025-04-30T06:46:24.101ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90392025-04-30T06:46:24.101ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90402025-04-30T06:46:24.102ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90412025-04-30T06:46:24.103ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90422025-04-30T06:46:24.116ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90432025-04-30T06:46:24.125ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90442025-04-30T06:46:24.126ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90452025-04-30T06:46:24.126ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
90462025-04-30T06:46:24.127ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90472025-04-30T06:46:24.127ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90482025-04-30T06:46:24.132ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90492025-04-30T06:46:24.132ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90502025-04-30T06:46:24.133ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90512025-04-30T06:46:24.133ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90522025-04-30T06:46:24.133ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90532025-04-30T06:46:24.133ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90542025-04-30T06:46:24.136ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90552025-04-30T06:46:24.140ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90562025-04-30T06:46:24.140ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90572025-04-30T06:46:24.140ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90582025-04-30T06:46:24.141ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90592025-04-30T06:46:24.167ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90602025-04-30T06:46:24.181ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90612025-04-30T06:46:24.181ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90622025-04-30T06:46:24.181ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
90632025-04-30T06:46:24.181ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90642025-04-30T06:46:24.276ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
90652025-04-30T06:46:24.276ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
90662025-04-30T06:46:24.279ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:46725 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90672025-04-30T06:46:24.280ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:46725 req_id = dff111f2-1d17-401a-a55a-8d102c82e399 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90682025-04-30T06:46:24.280ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 680 local_addr = [::1]:54590 method = GET remote_addr = [::1]:46725 req_id = dff111f2-1d17-401a-a55a-8d102c82e399 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90692025-04-30T06:46:24.281ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90702025-04-30T06:46:24.281ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
90712025-04-30T06:46:24.281ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90722025-04-30T06:46:24.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff111f2-1d17-401a-a55a-8d102c82e399", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
90732025-04-30T06:46:24.284ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90742025-04-30T06:46:24.284ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90752025-04-30T06:46:24.284ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
90762025-04-30T06:46:24.288ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90772025-04-30T06:46:24.288ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90782025-04-30T06:46:24.288ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90792025-04-30T06:46:24.288ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90802025-04-30T06:46:24.289ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90812025-04-30T06:46:24.297ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90822025-04-30T06:46:24.304ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90832025-04-30T06:46:24.305ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90842025-04-30T06:46:24.305ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90852025-04-30T06:46:24.305ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90862025-04-30T06:46:24.371ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
90872025-04-30T06:46:24.386ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90882025-04-30T06:46:24.386ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90892025-04-30T06:46:24.386ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 736.83762ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90902025-04-30T06:46:24.387ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90912025-04-30T06:46:24.387ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90922025-04-30T06:46:24.387ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90932025-04-30T06:46:24.387ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90942025-04-30T06:46:24.388ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90952025-04-30T06:46:24.389ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90962025-04-30T06:46:24.392ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90972025-04-30T06:46:24.396ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90982025-04-30T06:46:24.396ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90992025-04-30T06:46:24.396ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91002025-04-30T06:46:24.397ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91012025-04-30T06:46:24.402ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
91022025-04-30T06:46:24.402ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
91032025-04-30T06:46:24.403ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4f 1d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 O............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60536 req_id = dcbf24d2-77f0-48d7-8e17-222bf110fbad
91042025-04-30T06:46:24.403ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20253,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60536 req_id = dcbf24d2-77f0-48d7-8e17-222bf110fbad
91052025-04-30T06:46:24.404ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
91062025-04-30T06:46:24.404ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91072025-04-30T06:46:24.404ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60536 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dcbf24d2-77f0-48d7-8e17-222bf110fbad
91082025-04-30T06:46:24.404ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4449, tv_nsec: 203706008 } })
91092025-04-30T06:46:24.404ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
91102025-04-30T06:46:24.405ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 26 cf 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e &...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57402 req_id = 7f38595d-af30-490a-955a-f0e675c720ee
91112025-04-30T06:46:24.405ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9935,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57402 req_id = 7f38595d-af30-490a-955a-f0e675c720ee
91122025-04-30T06:46:24.405ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91132025-04-30T06:46:24.405ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57402 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7f38595d-af30-490a-955a-f0e675c720ee
91142025-04-30T06:46:24.529ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
91152025-04-30T06:46:24.530ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91162025-04-30T06:46:24.530ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91172025-04-30T06:46:24.530ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91182025-04-30T06:46:24.530ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91192025-04-30T06:46:24.531ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91202025-04-30T06:46:24.532ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 89 e1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59931 req_id = 77404da1-6fed-4f41-8b81-8d32af4c447c
91212025-04-30T06:46:24.532ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35297,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59931 req_id = 77404da1-6fed-4f41-8b81-8d32af4c447c
91222025-04-30T06:46:24.532ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
91232025-04-30T06:46:24.533ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35297, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59931 req_id = 77404da1-6fed-4f41-8b81-8d32af4c447c
91242025-04-30T06:46:24.533ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
91252025-04-30T06:46:24.533ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
91262025-04-30T06:46:24.533ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
91272025-04-30T06:46:24.534ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91282025-04-30T06:46:24.572ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91292025-04-30T06:46:24.572ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91302025-04-30T06:46:24.572ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91312025-04-30T06:46:24.573ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91322025-04-30T06:46:24.592ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91332025-04-30T06:46:24.592ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91342025-04-30T06:46:24.593ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91352025-04-30T06:46:24.593ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "edb092b6-a84e-42b5-81de-f26c60e53e34", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91362025-04-30T06:46:24.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
91372025-04-30T06:46:24.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17339283-37f6-4f5a-8f0c-50758b71c9cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91382025-04-30T06:46:24.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
91392025-04-30T06:46:24.632ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7892dbc-5cd3-41ec-8706-ad0e1343ad7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91402025-04-30T06:46:24.632ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
91412025-04-30T06:46:24.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91422025-04-30T06:46:24.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91432025-04-30T06:46:24.638ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91442025-04-30T06:46:24.639ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91452025-04-30T06:46:24.639ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91462025-04-30T06:46:24.639ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
91472025-04-30T06:46:24.640ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5b2b6dd-2fe2-4318-86f2-9a6251292c1a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91482025-04-30T06:46:24.641ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
91492025-04-30T06:46:24.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f6a3ac0-8d8e-4864-86a5-08915df1655b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91502025-04-30T06:46:24.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
91512025-04-30T06:46:24.653ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
91522025-04-30T06:46:24.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db53beff-24ca-4c29-8faa-3bfff676709d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91532025-04-30T06:46:24.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
91542025-04-30T06:46:24.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "921de142-2cc0-4034-a0c7-1c36103fe663", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91552025-04-30T06:46:24.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
91562025-04-30T06:46:24.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c951f90-b855-4350-b940-f1b9acba1e9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91572025-04-30T06:46:24.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
91582025-04-30T06:46:24.675ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0f25ca3-15d6-4616-b330-4be4b403d1de", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91592025-04-30T06:46:24.675ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
91602025-04-30T06:46:24.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce18594b-a1a6-4763-bbe2-106cf7f05705", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91612025-04-30T06:46:24.676ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
91622025-04-30T06:46:24.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4834170-3da9-478a-8591-f338f86a7731", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91632025-04-30T06:46:24.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
91642025-04-30T06:46:24.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f280d4b-d8fe-47dd-872b-09f3925f5dee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91652025-04-30T06:46:24.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
91662025-04-30T06:46:24.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b524d11-fb86-4e61-bd32-f81e4ce14581", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91672025-04-30T06:46:24.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
91682025-04-30T06:46:24.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfafea1a-64ca-4b09-8b79-d0485495ca0d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91692025-04-30T06:46:24.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
91702025-04-30T06:46:24.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff84a04-afec-4bd8-b731-75b0d16fd760", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91712025-04-30T06:46:24.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
91722025-04-30T06:46:24.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "172278a6-634c-49e5-835d-14d63c25653d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91732025-04-30T06:46:24.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
91742025-04-30T06:46:24.705ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91752025-04-30T06:46:24.705ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91762025-04-30T06:46:24.705ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91772025-04-30T06:46:24.705ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91782025-04-30T06:46:24.706ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91792025-04-30T06:46:24.710ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91802025-04-30T06:46:24.715ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91812025-04-30T06:46:24.715ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91822025-04-30T06:46:24.715ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91832025-04-30T06:46:24.716ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91842025-04-30T06:46:24.720ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91852025-04-30T06:46:24.720ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
91862025-04-30T06:46:24.721ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91872025-04-30T06:46:24.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32928810-bf34-4cf5-b7e9-e0ddcf23996b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91882025-04-30T06:46:24.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
91892025-04-30T06:46:24.735ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d6e117c-303a-4f0c-8cdf-14def1552469", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91902025-04-30T06:46:24.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
91912025-04-30T06:46:24.740ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3acb7c20-a15b-4140-936e-80c80d0b24d8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91922025-04-30T06:46:24.740ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
91932025-04-30T06:46:24.744ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe709499-b5e1-4be3-a145-cf424b122a93", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91942025-04-30T06:46:24.745ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
91952025-04-30T06:46:24.747ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d730918c-48f1-4225-ac37-3f7bf5347ef2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91962025-04-30T06:46:24.747ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
91972025-04-30T06:46:24.748ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c1eeb4a-4129-4939-9139-717111140c78", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
91982025-04-30T06:46:24.748ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
91992025-04-30T06:46:24.750ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a40a37b3-6ced-43a7-b70d-72f04474d4c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92002025-04-30T06:46:24.750ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
92012025-04-30T06:46:24.752ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0984aaf6-c084-4110-8e46-a66db82ca5aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92022025-04-30T06:46:24.752ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
92032025-04-30T06:46:24.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41b88867-8b42-4112-ba56-65e775917d34", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92042025-04-30T06:46:24.755ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
92052025-04-30T06:46:24.756ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "039d1190-4180-4312-963a-1644f18d4bbc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92062025-04-30T06:46:24.756ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
92072025-04-30T06:46:24.758ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f271fb58-0152-410f-9325-15de3db2e627", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92082025-04-30T06:46:24.758ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
92092025-04-30T06:46:24.760ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a56ff588-bdbc-4743-bc5f-bbd31fec09a8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92102025-04-30T06:46:24.760ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
92112025-04-30T06:46:24.762ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23cffccc-047f-439b-91ee-be5b1d18997f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92122025-04-30T06:46:24.763ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
92132025-04-30T06:46:24.766ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af97df00-f6ee-43ef-97bc-9180657e8896", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92142025-04-30T06:46:24.766ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
92152025-04-30T06:46:24.766ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5670d7fa-aaaf-49e1-b907-1b44adfbcdcb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92162025-04-30T06:46:24.767ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
92172025-04-30T06:46:24.772ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6956a8ef-780d-4584-acd3-8d155ad3b83d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92182025-04-30T06:46:24.802ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
92192025-04-30T06:46:24.807ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
92202025-04-30T06:46:24.809ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "044349f9-bb46-4333-a57b-bab1720d80d6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92212025-04-30T06:46:24.809ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
92222025-04-30T06:46:24.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef69db7-85cb-47d0-833e-7638fabe3243", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92232025-04-30T06:46:24.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
92242025-04-30T06:46:24.813ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6acda363-7da7-479a-99e6-fdde479212ae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92252025-04-30T06:46:24.813ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
92262025-04-30T06:46:24.816ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39e51a12-ea5e-42e1-a8c0-2fbc43a27eb0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92272025-04-30T06:46:24.816ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
92282025-04-30T06:46:24.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9599e35d-3e0b-44bf-a095-2431323e8caa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92292025-04-30T06:46:24.820ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
92302025-04-30T06:46:24.826ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7dfedee-3f22-4bb9-88e6-5270dbc17b6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92312025-04-30T06:46:24.826ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
92322025-04-30T06:46:24.830ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a604c2-4851-4d48-9e76-41384b7f1377", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92332025-04-30T06:46:24.830ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
92342025-04-30T06:46:24.833ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "624a2695-0f7e-4545-b7ee-7831d5e05f94", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92352025-04-30T06:46:24.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
92362025-04-30T06:46:24.858ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92372025-04-30T06:46:24.858ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92382025-04-30T06:46:24.858ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92392025-04-30T06:46:24.858ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
92402025-04-30T06:46:24.860ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92412025-04-30T06:46:24.864ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92422025-04-30T06:46:24.872ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92432025-04-30T06:46:24.872ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92442025-04-30T06:46:24.872ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
92452025-04-30T06:46:24.872ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92462025-04-30T06:46:24.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "674437d0-6bb3-49aa-a62a-c4265b89f667", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92472025-04-30T06:46:24.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
92482025-04-30T06:46:24.876ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92492025-04-30T06:46:24.876ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
92502025-04-30T06:46:24.877ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92512025-04-30T06:46:24.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6296d135-6039-4c74-9344-27e79d14b1c2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92522025-04-30T06:46:24.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
92532025-04-30T06:46:24.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87813c94-2c25-4858-a4f3-ff7d2b6c6f8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92542025-04-30T06:46:24.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
92552025-04-30T06:46:24.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f264be2-179f-49bb-9454-42298f033962", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92562025-04-30T06:46:24.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
92572025-04-30T06:46:24.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a633bd6-ae99-4a86-a4c0-bc1ab60e1a0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92582025-04-30T06:46:24.889ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92592025-04-30T06:46:24.890ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92602025-04-30T06:46:24.890ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92612025-04-30T06:46:24.890ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92622025-04-30T06:46:24.891ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92632025-04-30T06:46:24.902ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92642025-04-30T06:46:24.916ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92652025-04-30T06:46:24.916ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92662025-04-30T06:46:24.916ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92672025-04-30T06:46:24.916ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92682025-04-30T06:46:24.916ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92692025-04-30T06:46:24.917ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92702025-04-30T06:46:24.929ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92712025-04-30T06:46:24.935ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92722025-04-30T06:46:24.935ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92732025-04-30T06:46:24.935ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92742025-04-30T06:46:24.936ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92752025-04-30T06:46:25.052ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
92762025-04-30T06:46:25.053ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
92772025-04-30T06:46:25.054ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:39713 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92782025-04-30T06:46:25.055ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
92792025-04-30T06:46:25.055ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
92802025-04-30T06:46:25.057ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
92812025-04-30T06:46:25.058ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
92822025-04-30T06:46:25.059ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:39713 req_id = 5e8a4e20-aec5-4b48-80f7-e14a1909401b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92832025-04-30T06:46:25.059ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 559 local_addr = [::1]:54590 method = GET remote_addr = [::1]:39713 req_id = 5e8a4e20-aec5-4b48-80f7-e14a1909401b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92842025-04-30T06:46:25.060ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e8a4e20-aec5-4b48-80f7-e14a1909401b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:24 GMT"} })
92852025-04-30T06:46:25.062ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92862025-04-30T06:46:25.062ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92872025-04-30T06:46:25.062ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
92882025-04-30T06:46:25.065ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92892025-04-30T06:46:25.065ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92902025-04-30T06:46:25.065ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92912025-04-30T06:46:25.065ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92922025-04-30T06:46:25.067ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92932025-04-30T06:46:25.072ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
92942025-04-30T06:46:25.072ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
92952025-04-30T06:46:25.073ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92962025-04-30T06:46:25.080ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92972025-04-30T06:46:25.080ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92982025-04-30T06:46:25.081ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92992025-04-30T06:46:25.081ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93002025-04-30T06:46:25.083ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
93012025-04-30T06:46:25.089ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
93022025-04-30T06:46:25.089ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93032025-04-30T06:46:25.136ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93042025-04-30T06:46:25.136ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93052025-04-30T06:46:25.136ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93062025-04-30T06:46:25.136ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
93072025-04-30T06:46:25.137ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93082025-04-30T06:46:25.155ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93092025-04-30T06:46:25.162ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93102025-04-30T06:46:25.162ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93112025-04-30T06:46:25.162ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
93122025-04-30T06:46:25.163ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93132025-04-30T06:46:25.164ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
93142025-04-30T06:46:25.164ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
93152025-04-30T06:46:25.166ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
93162025-04-30T06:46:25.167ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 779.175637ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
93172025-04-30T06:46:25.167ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
93182025-04-30T06:46:25.167ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93192025-04-30T06:46:25.167ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93202025-04-30T06:46:25.167ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93212025-04-30T06:46:25.167ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93222025-04-30T06:46:25.167ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93232025-04-30T06:46:25.179ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93242025-04-30T06:46:25.179ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
93252025-04-30T06:46:25.179ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93262025-04-30T06:46:25.219ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93272025-04-30T06:46:25.223ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93282025-04-30T06:46:25.223ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93292025-04-30T06:46:25.223ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93302025-04-30T06:46:25.224ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93312025-04-30T06:46:25.231ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
93322025-04-30T06:46:25.231ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93332025-04-30T06:46:25.233ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 01 b4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61340 req_id = e60cbfe8-9466-4652-b6cd-d379bb1f7436
93342025-04-30T06:46:25.233ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 436,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61340 req_id = e60cbfe8-9466-4652-b6cd-d379bb1f7436
93352025-04-30T06:46:25.234ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
93362025-04-30T06:46:25.234ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93372025-04-30T06:46:25.235ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61340 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e60cbfe8-9466-4652-b6cd-d379bb1f7436
93382025-04-30T06:46:25.237ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4450, tv_nsec: 35933817 } })
93392025-04-30T06:46:25.237ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
93402025-04-30T06:46:25.238ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 80 0c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62089 req_id = 6e887bbd-1d3a-41ca-af40-4a0441506d79
93412025-04-30T06:46:25.238ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32780,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62089 req_id = 6e887bbd-1d3a-41ca-af40-4a0441506d79
93422025-04-30T06:46:25.238ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93432025-04-30T06:46:25.239ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62089 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6e887bbd-1d3a-41ca-af40-4a0441506d79
93442025-04-30T06:46:25.295ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93452025-04-30T06:46:25.295ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5f 07 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c _............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60001 req_id = 6791b355-f6f4-488e-b33c-73f51e9d9af8
93462025-04-30T06:46:25.296ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24327,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60001 req_id = 6791b355-f6f4-488e-b33c-73f51e9d9af8
93472025-04-30T06:46:25.296ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
93482025-04-30T06:46:25.296ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24327, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60001 req_id = 6791b355-f6f4-488e-b33c-73f51e9d9af8
93492025-04-30T06:46:25.297ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
93502025-04-30T06:46:25.297ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
93512025-04-30T06:46:25.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
93522025-04-30T06:46:25.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7e3df27-6138-4f54-971b-e26f436f7443", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93532025-04-30T06:46:25.300ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
93542025-04-30T06:46:25.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c28cc43a-bf32-4fb5-8a3f-fc9e023c7d54", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93552025-04-30T06:46:25.302ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
93562025-04-30T06:46:25.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bfb3bd9-e988-43f4-8a5d-fcd0dcc3776f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93572025-04-30T06:46:25.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
93582025-04-30T06:46:25.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6537f83-ffa5-4d84-b07d-b0a9ab81c9e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93592025-04-30T06:46:25.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
93602025-04-30T06:46:25.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "087cbb88-83d5-4979-8ba2-a9105c172a5e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93612025-04-30T06:46:25.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
93622025-04-30T06:46:25.315ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74b402de-b857-4e1b-b8b4-43471d8c4576", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93632025-04-30T06:46:25.316ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
93642025-04-30T06:46:25.316ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
93652025-04-30T06:46:25.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5b9d95-55b3-4b90-b999-daf557302a76", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93662025-04-30T06:46:25.319ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
93672025-04-30T06:46:25.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd318ea-7083-4cbf-8389-0e1fc096f1f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93682025-04-30T06:46:25.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
93692025-04-30T06:46:25.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24368047-e9ac-47b3-a5ce-31d21b20d0ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93702025-04-30T06:46:25.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
93712025-04-30T06:46:25.323ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43337daf-8ae4-4ce0-8c18-e6ed799b35ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93722025-04-30T06:46:25.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
93732025-04-30T06:46:25.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "793b7965-ec1d-4f08-a9dc-a9fb492cf1bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93742025-04-30T06:46:25.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
93752025-04-30T06:46:25.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5383ac5-06d9-452f-991a-fafecd9b1d53", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93762025-04-30T06:46:25.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
93772025-04-30T06:46:25.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f18de86-b0e0-4046-93c9-38496050d920", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93782025-04-30T06:46:25.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
93792025-04-30T06:46:25.330ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65ba484-bd30-4937-a76e-079c3570675f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93802025-04-30T06:46:25.330ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
93812025-04-30T06:46:25.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "904b1c69-c170-495d-a0e4-ff01b97835f7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93822025-04-30T06:46:25.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
93832025-04-30T06:46:25.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e34aa89-37f2-4ab1-8caa-13fb86ae9f6a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93842025-04-30T06:46:25.337ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
93852025-04-30T06:46:25.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de69034-eba2-4922-9e8e-8d9dcc5ecba2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93862025-04-30T06:46:25.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
93872025-04-30T06:46:25.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65031312-e14e-48dc-a174-9703932442b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93882025-04-30T06:46:25.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
93892025-04-30T06:46:25.373ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93902025-04-30T06:46:25.373ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93912025-04-30T06:46:25.373ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93922025-04-30T06:46:25.373ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
93932025-04-30T06:46:25.374ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93942025-04-30T06:46:25.374ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a90a4b-9c90-4191-a180-b7485e3b1377", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93952025-04-30T06:46:25.374ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
93962025-04-30T06:46:25.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfec4375-22fd-4169-841a-9b70fffa07f6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
93972025-04-30T06:46:25.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
93982025-04-30T06:46:25.385ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93992025-04-30T06:46:25.390ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94002025-04-30T06:46:25.390ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94012025-04-30T06:46:25.390ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
94022025-04-30T06:46:25.391ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94032025-04-30T06:46:25.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d602b65b-1be0-4735-ac3b-2202b02d4d2f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94042025-04-30T06:46:25.391ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
94052025-04-30T06:46:25.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "758142fe-2734-40d9-a19f-1a9bae74bcb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94062025-04-30T06:46:25.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
94072025-04-30T06:46:25.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b862bbca-0cc7-4289-a82f-5d8fdef62061", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94082025-04-30T06:46:25.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
94092025-04-30T06:46:25.394ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94102025-04-30T06:46:25.394ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
94112025-04-30T06:46:25.394ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94122025-04-30T06:46:25.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad2b4735-7ade-4c47-9e17-e8f87f76b2dc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94132025-04-30T06:46:25.396ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
94142025-04-30T06:46:25.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be0f8e20-4242-4359-ae27-37f3d10e8fe7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94152025-04-30T06:46:25.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
94162025-04-30T06:46:25.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b02365a-ff3e-470a-8a67-54210a3365b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94172025-04-30T06:46:25.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
94182025-04-30T06:46:25.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bedecef6-ef08-47f1-bd97-3d5dab77b999", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94192025-04-30T06:46:25.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
94202025-04-30T06:46:25.418ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c4c370e-bf22-40df-ba0b-f7cf27a8eec7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94212025-04-30T06:46:25.418ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
94222025-04-30T06:46:25.436ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
94232025-04-30T06:46:25.446ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19f4cece-d8dc-4be1-a66e-09b4e5caaf28", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94242025-04-30T06:46:25.446ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
94252025-04-30T06:46:25.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "28766245-3478-4210-aa5f-4bd32f59c631", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94262025-04-30T06:46:25.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
94272025-04-30T06:46:25.451ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09c1243a-8631-429e-82ed-283e5c32643f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94282025-04-30T06:46:25.451ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
94292025-04-30T06:46:25.452ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e0c367c-cc1d-4074-ba1c-a72084cab42f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94302025-04-30T06:46:25.453ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
94312025-04-30T06:46:25.454ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44eb1282-4be9-4f7a-920a-cc06e92f3327", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94322025-04-30T06:46:25.454ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
94332025-04-30T06:46:25.455ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7d064f-5929-4277-9dd7-ca78f6922695", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94342025-04-30T06:46:25.455ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
94352025-04-30T06:46:25.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41173b28-f776-4de6-8be1-2163e862ec2b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94362025-04-30T06:46:25.457ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
94372025-04-30T06:46:25.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb71e92-44f2-498c-a614-75a1454b9cfe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94382025-04-30T06:46:25.459ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
94392025-04-30T06:46:25.461ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2362bcd1-c0e1-47ee-b4e5-0dd9527135ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94402025-04-30T06:46:25.462ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
94412025-04-30T06:46:25.464ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e284811-1005-4887-95ff-c9291dab24ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94422025-04-30T06:46:25.464ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
94432025-04-30T06:46:25.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df84bca4-a33a-4509-8643-44da4fc44e4b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94442025-04-30T06:46:25.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
94452025-04-30T06:46:25.469ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2029bf8c-689d-4ece-86fe-237944a395c5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94462025-04-30T06:46:25.469ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
94472025-04-30T06:46:25.471ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "896ccdaa-6151-4316-a246-a36536e03325", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94482025-04-30T06:46:25.471ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
94492025-04-30T06:46:25.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e37126-6e8c-43f5-b4e8-3a32d4107203", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94502025-04-30T06:46:25.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
94512025-04-30T06:46:25.473ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1b6248-b187-47bd-9e02-f412a068a7fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94522025-04-30T06:46:25.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
94532025-04-30T06:46:25.475ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2cdc587-9854-4203-8824-2c9e3de90615", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94542025-04-30T06:46:25.475ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
94552025-04-30T06:46:25.476ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32025eaf-e592-4cdc-bcf2-3627a41d7e3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94562025-04-30T06:46:25.476ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94572025-04-30T06:46:25.476ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94582025-04-30T06:46:25.477ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94592025-04-30T06:46:25.477ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94602025-04-30T06:46:25.479ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94612025-04-30T06:46:25.483ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94622025-04-30T06:46:25.489ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94632025-04-30T06:46:25.489ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94642025-04-30T06:46:25.489ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94652025-04-30T06:46:25.489ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94662025-04-30T06:46:25.489ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94672025-04-30T06:46:25.490ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94682025-04-30T06:46:25.491ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94692025-04-30T06:46:25.491ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94702025-04-30T06:46:25.491ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94712025-04-30T06:46:25.491ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
94722025-04-30T06:46:25.492ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94732025-04-30T06:46:25.496ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94742025-04-30T06:46:25.502ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94752025-04-30T06:46:25.502ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94762025-04-30T06:46:25.502ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94772025-04-30T06:46:25.503ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94782025-04-30T06:46:25.506ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94792025-04-30T06:46:25.512ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94802025-04-30T06:46:25.512ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94812025-04-30T06:46:25.512ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
94822025-04-30T06:46:25.512ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94832025-04-30T06:46:25.596ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
94842025-04-30T06:46:25.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
94852025-04-30T06:46:25.597ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:55917 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94862025-04-30T06:46:25.597ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
94872025-04-30T06:46:25.598ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94882025-04-30T06:46:25.599ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94892025-04-30T06:46:25.599ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
94902025-04-30T06:46:25.601ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94912025-04-30T06:46:25.601ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f2293fc6-10ef-48df-9542-fd25ba333b1f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94922025-04-30T06:46:25.601ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:55917 req_id = 40648f8e-77f6-4b4b-97ad-632fdfff4e40 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94932025-04-30T06:46:25.602ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f2293fc6-10ef-48df-9542-fd25ba333b1f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94942025-04-30T06:46:25.602ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f2293fc6-10ef-48df-9542-fd25ba333b1f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94952025-04-30T06:46:25.602ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f2293fc6-10ef-48df-9542-fd25ba333b1f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94962025-04-30T06:46:25.603ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2149 local_addr = [::1]:54590 method = GET remote_addr = [::1]:55917 req_id = 40648f8e-77f6-4b4b-97ad-632fdfff4e40 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94972025-04-30T06:46:25.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "40648f8e-77f6-4b4b-97ad-632fdfff4e40", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
94982025-04-30T06:46:25.606ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94992025-04-30T06:46:25.606ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95002025-04-30T06:46:25.606ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
95012025-04-30T06:46:25.611ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95022025-04-30T06:46:25.611ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95032025-04-30T06:46:25.611ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95042025-04-30T06:46:25.611ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95052025-04-30T06:46:25.612ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95062025-04-30T06:46:25.615ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 14513 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f2293fc6-10ef-48df-9542-fd25ba333b1f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95072025-04-30T06:46:25.616ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2293fc6-10ef-48df-9542-fd25ba333b1f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95082025-04-30T06:46:25.616ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95092025-04-30T06:46:25.616ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95102025-04-30T06:46:25.623ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95112025-04-30T06:46:25.623ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95122025-04-30T06:46:25.623ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95132025-04-30T06:46:25.624ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95142025-04-30T06:46:25.625ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 83fff29f-5f2e-4811-8b5d-76d7d18264d9 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95152025-04-30T06:46:25.625ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 83fff29f-5f2e-4811-8b5d-76d7d18264d9 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95162025-04-30T06:46:25.625ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 83fff29f-5f2e-4811-8b5d-76d7d18264d9 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95172025-04-30T06:46:25.625ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 83fff29f-5f2e-4811-8b5d-76d7d18264d9 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95182025-04-30T06:46:25.631ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6709 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 83fff29f-5f2e-4811-8b5d-76d7d18264d9 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95192025-04-30T06:46:25.634ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "83fff29f-5f2e-4811-8b5d-76d7d18264d9", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95202025-04-30T06:46:25.636ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
95212025-04-30T06:46:25.637ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95222025-04-30T06:46:25.637ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
95232025-04-30T06:46:25.637ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95242025-04-30T06:46:25.637ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95252025-04-30T06:46:25.640ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95262025-04-30T06:46:25.640ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95272025-04-30T06:46:25.641ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95282025-04-30T06:46:25.641ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95292025-04-30T06:46:25.666ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
95302025-04-30T06:46:25.718ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95312025-04-30T06:46:25.718ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95322025-04-30T06:46:25.718ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95332025-04-30T06:46:25.718ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
95342025-04-30T06:46:25.718ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95352025-04-30T06:46:25.735ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95362025-04-30T06:46:25.741ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95372025-04-30T06:46:25.741ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95382025-04-30T06:46:25.741ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
95392025-04-30T06:46:25.741ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95402025-04-30T06:46:25.760ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
95412025-04-30T06:46:25.760ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
95422025-04-30T06:46:25.760ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 593.662915ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
95432025-04-30T06:46:25.760ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
95442025-04-30T06:46:25.760ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95452025-04-30T06:46:25.760ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95462025-04-30T06:46:25.760ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95472025-04-30T06:46:25.760ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95482025-04-30T06:46:25.761ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95492025-04-30T06:46:25.763ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95502025-04-30T06:46:25.766ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95512025-04-30T06:46:25.767ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95522025-04-30T06:46:25.767ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95532025-04-30T06:46:25.767ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95542025-04-30T06:46:25.767ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95552025-04-30T06:46:25.767ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
95562025-04-30T06:46:25.768ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95572025-04-30T06:46:25.770ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
95582025-04-30T06:46:25.770ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95592025-04-30T06:46:25.771ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c0 9e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63659 req_id = 927058a9-4dc4-4bbb-94b4-962410737977
95602025-04-30T06:46:25.771ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49310,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63659 req_id = 927058a9-4dc4-4bbb-94b4-962410737977
95612025-04-30T06:46:25.771ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
95622025-04-30T06:46:25.771ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95632025-04-30T06:46:25.771ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63659 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 927058a9-4dc4-4bbb-94b4-962410737977
95642025-04-30T06:46:25.772ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4450, tv_nsec: 571407253 } })
95652025-04-30T06:46:25.772ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
95662025-04-30T06:46:25.774ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d9 1d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55071 req_id = 10c11db4-ad04-49ad-ba19-b0d3dfeff6b8
95672025-04-30T06:46:25.774ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55581,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55071 req_id = 10c11db4-ad04-49ad-ba19-b0d3dfeff6b8
95682025-04-30T06:46:25.774ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95692025-04-30T06:46:25.774ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55071 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 10c11db4-ad04-49ad-ba19-b0d3dfeff6b8
95702025-04-30T06:46:25.824ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95712025-04-30T06:46:25.825ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 78 52 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c xR..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56348 req_id = cae69791-0e62-4d2c-8510-68da031e8e7d
95722025-04-30T06:46:25.825ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30802,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56348 req_id = cae69791-0e62-4d2c-8510-68da031e8e7d
95732025-04-30T06:46:25.825ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
95742025-04-30T06:46:25.825ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30802, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56348 req_id = cae69791-0e62-4d2c-8510-68da031e8e7d
95752025-04-30T06:46:25.825ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
95762025-04-30T06:46:25.825ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
95772025-04-30T06:46:25.825ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
95782025-04-30T06:46:25.827ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7fefc7-dd1a-462f-9c0c-01fe8493699d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95792025-04-30T06:46:25.827ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
95802025-04-30T06:46:25.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2cad80b-0568-4a39-b6f5-e40479cda492", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95812025-04-30T06:46:25.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
95822025-04-30T06:46:25.831ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f1be5df-c57f-45a5-8ec5-fd67004fb536", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95832025-04-30T06:46:25.832ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
95842025-04-30T06:46:25.834ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80474cb4-21ff-4bc5-8f0e-f2a43741fcd7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95852025-04-30T06:46:25.834ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
95862025-04-30T06:46:25.838ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41cba0e0-e7ad-4773-a213-47c90a7b78c9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95872025-04-30T06:46:25.838ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
95882025-04-30T06:46:25.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f98695cc-b5b5-479f-87b2-54ab22d30462", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95892025-04-30T06:46:25.841ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
95902025-04-30T06:46:25.844ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37d53fbd-c9fd-4daa-ab51-8405b1f3e61b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95912025-04-30T06:46:25.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
95922025-04-30T06:46:25.849ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
95932025-04-30T06:46:25.850ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da989217-f8a7-4727-96c1-874dc828aea1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95942025-04-30T06:46:25.850ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
95952025-04-30T06:46:25.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10dec3c8-9c42-45e6-91f6-58341bfae2e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95962025-04-30T06:46:25.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
95972025-04-30T06:46:25.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f0e8d21-1528-428a-94c7-681e5068d298", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
95982025-04-30T06:46:25.853ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
95992025-04-30T06:46:25.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d5a0317-6a9a-42c6-83f4-0f53ecbe1dd2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96002025-04-30T06:46:25.855ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
96012025-04-30T06:46:25.857ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb4efee8-e9b9-45cc-9c0f-2aae827dc322", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96022025-04-30T06:46:25.858ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
96032025-04-30T06:46:25.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40c12872-78fd-4e26-81e6-440c188dd3df", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96042025-04-30T06:46:25.860ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
96052025-04-30T06:46:25.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a38afe79-6b05-4cb6-a100-15097822860e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96062025-04-30T06:46:25.864ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
96072025-04-30T06:46:25.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e895eec9-afe1-4d01-9923-2c1ccd1edc9a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96082025-04-30T06:46:25.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
96092025-04-30T06:46:25.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "912c0ac7-9a50-4386-9987-7303167ed7bb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96102025-04-30T06:46:25.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
96112025-04-30T06:46:25.872ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1a91c4-3706-4d7b-8954-a25184ea8bb6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96122025-04-30T06:46:25.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
96132025-04-30T06:46:25.874ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a5c9c3-16ba-431a-9b0e-59935d52dfac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96142025-04-30T06:46:25.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
96152025-04-30T06:46:25.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4139ee83-8bd2-4ba2-8168-fa2156e05830", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96162025-04-30T06:46:25.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
96172025-04-30T06:46:25.878ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "135ea776-829b-4a94-ae56-a7a8fec4ba48", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96182025-04-30T06:46:25.878ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
96192025-04-30T06:46:25.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9db41f8-16bc-4309-aac7-8f224d65b3a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96202025-04-30T06:46:25.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
96212025-04-30T06:46:25.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12653063-bc30-4c62-88da-f71e3ecc4ebc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96222025-04-30T06:46:25.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
96232025-04-30T06:46:25.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e6195d6-6935-4eb9-9aaa-179889fbf62d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96242025-04-30T06:46:25.882ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
96252025-04-30T06:46:25.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52b55887-5b66-4416-a850-c6c97acd6c41", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96262025-04-30T06:46:25.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
96272025-04-30T06:46:25.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45f5588e-f7cc-4fb4-bb94-26faf977d3e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96282025-04-30T06:46:25.886ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
96292025-04-30T06:46:25.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60601900-6cbe-440d-adb7-62f881a3923b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96302025-04-30T06:46:25.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
96312025-04-30T06:46:25.890ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c333857-0ab2-424f-97ed-56f8b1e4a9cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96322025-04-30T06:46:25.890ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
96332025-04-30T06:46:25.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96e7678-e7df-4acb-b31c-e2ec1134225a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96342025-04-30T06:46:25.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
96352025-04-30T06:46:25.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4196cb75-e6f1-4dc4-a6e6-9430dfd04dba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96362025-04-30T06:46:25.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
96372025-04-30T06:46:25.896ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "792756e5-1acb-4297-aa74-d02468c5299d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96382025-04-30T06:46:25.896ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
96392025-04-30T06:46:25.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80bf5b0c-a582-499b-ae1d-72ea98a9c03a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96402025-04-30T06:46:25.898ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
96412025-04-30T06:46:25.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f503fe72-1256-4feb-9160-d8348fbab66e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96422025-04-30T06:46:25.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
96432025-04-30T06:46:25.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8123d0-f936-4666-a303-45c9e6821d61", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96442025-04-30T06:46:25.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
96452025-04-30T06:46:25.900ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96462025-04-30T06:46:25.900ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96472025-04-30T06:46:25.901ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96482025-04-30T06:46:25.901ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
96492025-04-30T06:46:25.902ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96502025-04-30T06:46:25.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e806023-a669-4a41-8d8e-7d902434fcb3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96512025-04-30T06:46:25.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
96522025-04-30T06:46:25.904ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96532025-04-30T06:46:25.910ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96542025-04-30T06:46:25.910ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96552025-04-30T06:46:25.910ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
96562025-04-30T06:46:25.911ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96572025-04-30T06:46:25.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "088ba9e1-d64f-4a23-ae42-907d2386493b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96582025-04-30T06:46:25.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
96592025-04-30T06:46:25.914ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96602025-04-30T06:46:25.914ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
96612025-04-30T06:46:25.915ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96622025-04-30T06:46:25.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54768763-edef-4337-acaf-a80c78a1de00", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96632025-04-30T06:46:25.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
96642025-04-30T06:46:25.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43023e15-c1cc-44ad-8c9c-d42764214ffd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96652025-04-30T06:46:25.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
96662025-04-30T06:46:25.925ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0961ed6f-b07e-4fd8-a667-03d37ffcbf2f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96672025-04-30T06:46:25.925ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
96682025-04-30T06:46:25.928ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25a56be8-16d7-4a73-b180-9bc492471abd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96692025-04-30T06:46:25.928ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
96702025-04-30T06:46:25.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abe850b-36e4-40d8-99b2-be45a4d1ee3e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96712025-04-30T06:46:25.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
96722025-04-30T06:46:25.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb5dd3f-0a82-4254-944f-73706df1dec0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96732025-04-30T06:46:25.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
96742025-04-30T06:46:25.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61a2a5b2-838f-4de0-89e0-7eff2bf8e3dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96752025-04-30T06:46:25.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
96762025-04-30T06:46:25.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dff0dfc-1fa7-4bda-bd1d-c9a10074f814", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96772025-04-30T06:46:25.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
96782025-04-30T06:46:25.944ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07407fd6-5bea-4189-a37f-085bfc9221cc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96792025-04-30T06:46:25.945ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
96802025-04-30T06:46:25.946ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b7a2c1c-4df4-4a9c-85db-1b9405989a71", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
96812025-04-30T06:46:25.946ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96822025-04-30T06:46:25.946ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96832025-04-30T06:46:25.946ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96842025-04-30T06:46:25.946ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96852025-04-30T06:46:25.947ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96862025-04-30T06:46:25.949ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96872025-04-30T06:46:25.955ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96882025-04-30T06:46:25.955ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96892025-04-30T06:46:25.955ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96902025-04-30T06:46:25.955ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96912025-04-30T06:46:25.955ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96922025-04-30T06:46:25.955ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96932025-04-30T06:46:25.963ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
96942025-04-30T06:46:25.963ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96952025-04-30T06:46:25.968ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96962025-04-30T06:46:25.968ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96972025-04-30T06:46:25.968ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96982025-04-30T06:46:25.968ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96992025-04-30T06:46:26.027ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
97002025-04-30T06:46:26.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
97012025-04-30T06:46:26.028ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97022025-04-30T06:46:26.028ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97032025-04-30T06:46:26.028ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97042025-04-30T06:46:26.028ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97052025-04-30T06:46:26.029ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97062025-04-30T06:46:26.029ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:61194 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97072025-04-30T06:46:26.029ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97082025-04-30T06:46:26.030ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97092025-04-30T06:46:26.030ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
97102025-04-30T06:46:26.031ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
97112025-04-30T06:46:26.032ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:61194 req_id = de2e7fa0-19ea-49fb-83a6-123df7735188 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97122025-04-30T06:46:26.032ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 527 local_addr = [::1]:54590 method = GET remote_addr = [::1]:61194 req_id = de2e7fa0-19ea-49fb-83a6-123df7735188 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97132025-04-30T06:46:26.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2e7fa0-19ea-49fb-83a6-123df7735188", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97142025-04-30T06:46:26.034ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97152025-04-30T06:46:26.034ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97162025-04-30T06:46:26.034ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
97172025-04-30T06:46:26.036ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97182025-04-30T06:46:26.036ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97192025-04-30T06:46:26.036ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97202025-04-30T06:46:26.037ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97212025-04-30T06:46:26.037ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97222025-04-30T06:46:26.045ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97232025-04-30T06:46:26.052ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97242025-04-30T06:46:26.052ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97252025-04-30T06:46:26.052ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97262025-04-30T06:46:26.052ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97272025-04-30T06:46:26.053ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
97282025-04-30T06:46:26.053ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
97292025-04-30T06:46:26.057ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97302025-04-30T06:46:26.063ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97312025-04-30T06:46:26.063ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97322025-04-30T06:46:26.063ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97332025-04-30T06:46:26.064ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97342025-04-30T06:46:26.064ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
97352025-04-30T06:46:26.065ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97362025-04-30T06:46:26.065ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97372025-04-30T06:46:26.065ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97382025-04-30T06:46:26.066ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97392025-04-30T06:46:26.103ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97402025-04-30T06:46:26.103ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97412025-04-30T06:46:26.103ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 343.223691ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97422025-04-30T06:46:26.103ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97432025-04-30T06:46:26.103ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97442025-04-30T06:46:26.103ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97452025-04-30T06:46:26.103ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97462025-04-30T06:46:26.104ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97472025-04-30T06:46:26.104ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97482025-04-30T06:46:26.112ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97492025-04-30T06:46:26.116ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97502025-04-30T06:46:26.116ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97512025-04-30T06:46:26.116ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97522025-04-30T06:46:26.116ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97532025-04-30T06:46:26.117ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
97542025-04-30T06:46:26.122ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
97552025-04-30T06:46:26.122ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97562025-04-30T06:46:26.125ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: af 70 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .p..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51498 req_id = 8123985b-ab0c-42d3-ab28-d6b6945b4434
97572025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51498 req_id = 8123985b-ab0c-42d3-ab28-d6b6945b4434
97582025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
97592025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97602025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51498 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8123985b-ab0c-42d3-ab28-d6b6945b4434
97612025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4450, tv_nsec: 923621778 } })
97622025-04-30T06:46:26.125ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
97632025-04-30T06:46:26.125ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b9 89 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51304 req_id = 8d3e5de2-685e-4616-9d18-43beedec54e7
97642025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47497,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51304 req_id = 8d3e5de2-685e-4616-9d18-43beedec54e7
97652025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97662025-04-30T06:46:26.125ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51304 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8d3e5de2-685e-4616-9d18-43beedec54e7
97672025-04-30T06:46:26.172ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97682025-04-30T06:46:26.172ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97692025-04-30T06:46:26.172ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97702025-04-30T06:46:26.172ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97712025-04-30T06:46:26.172ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97722025-04-30T06:46:26.173ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97732025-04-30T06:46:26.173ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
97742025-04-30T06:46:26.174ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 32 78 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 2x..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55131 req_id = 5b21666d-f073-40be-9dde-3dffeab33141
97752025-04-30T06:46:26.174ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12920,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55131 req_id = 5b21666d-f073-40be-9dde-3dffeab33141
97762025-04-30T06:46:26.174ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
97772025-04-30T06:46:26.174ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55131 req_id = 5b21666d-f073-40be-9dde-3dffeab33141
97782025-04-30T06:46:26.174ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
97792025-04-30T06:46:26.174ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
97802025-04-30T06:46:26.174ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
97812025-04-30T06:46:26.176ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97822025-04-30T06:46:26.182ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97832025-04-30T06:46:26.183ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97842025-04-30T06:46:26.183ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97852025-04-30T06:46:26.183ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97862025-04-30T06:46:26.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3b0e220-16af-4b15-80b3-d9937129061a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97872025-04-30T06:46:26.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
97882025-04-30T06:46:26.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e81de33d-0e5c-4fe1-8888-899a0836edfc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97892025-04-30T06:46:26.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
97902025-04-30T06:46:26.186ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97912025-04-30T06:46:26.186ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
97922025-04-30T06:46:26.187ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97932025-04-30T06:46:26.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1ea3e01-3876-42dd-9b71-18f288bab36b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97942025-04-30T06:46:26.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
97952025-04-30T06:46:26.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a6a1b09-89a1-43e1-a769-f4587edc9e6c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97962025-04-30T06:46:26.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
97972025-04-30T06:46:26.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "021cc952-4ffe-4398-9f7e-737fd599d95b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
97982025-04-30T06:46:26.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
97992025-04-30T06:46:26.205ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3205f6f6-d862-4c6f-a311-2389b049980c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98002025-04-30T06:46:26.205ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
98012025-04-30T06:46:26.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f1f11f0-5661-43c1-86cb-cd38669e21d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98022025-04-30T06:46:26.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
98032025-04-30T06:46:26.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3dd8577-cfe5-4097-9087-7785636169f6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98042025-04-30T06:46:26.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
98052025-04-30T06:46:26.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e7a9bb5-08cb-4e0a-8e2a-b15ad64c15d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98062025-04-30T06:46:26.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
98072025-04-30T06:46:26.219ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
98082025-04-30T06:46:26.221ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6f7d39a-56fb-4860-8a01-6f047c3d6108", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98092025-04-30T06:46:26.221ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
98102025-04-30T06:46:26.222ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c22e3bcf-4783-4dfc-91b8-fca3959b9ea6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98112025-04-30T06:46:26.222ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
98122025-04-30T06:46:26.223ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f04ef63-0c71-4e05-9340-e86e71f75a48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98132025-04-30T06:46:26.223ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
98142025-04-30T06:46:26.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15965ba1-b597-4183-a3b2-9db8fd2a8ff2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98152025-04-30T06:46:26.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
98162025-04-30T06:46:26.227ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cdf4e2a-d44a-4b57-90f4-004d9ffb3878", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98172025-04-30T06:46:26.227ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
98182025-04-30T06:46:26.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e1d4947-a6d6-4ca2-a54f-59d906b371e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98192025-04-30T06:46:26.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
98202025-04-30T06:46:26.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc9ede2b-1576-4eee-bb44-ffa26da36397", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98212025-04-30T06:46:26.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
98222025-04-30T06:46:26.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffdf005-0a1f-42e3-9a8d-0378016d5613", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98232025-04-30T06:46:26.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
98242025-04-30T06:46:26.244ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e92cb479-72b5-4468-b73e-0348f44837d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98252025-04-30T06:46:26.244ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
98262025-04-30T06:46:26.246ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35de83e3-2519-44a6-9ac1-0826f2a0e60b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98272025-04-30T06:46:26.246ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
98282025-04-30T06:46:26.247ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "332ff900-b42f-46cd-87ed-3ef0fca69f9b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98292025-04-30T06:46:26.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
98302025-04-30T06:46:26.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0946184-0c6d-4c8f-b2c5-ade528b9e838", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98312025-04-30T06:46:26.248ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
98322025-04-30T06:46:26.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "561c925b-c0a7-4bd9-9a38-971345671307", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98332025-04-30T06:46:26.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
98342025-04-30T06:46:26.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0028764-5a3e-4575-bdfa-1e3f24ffcee1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98352025-04-30T06:46:26.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
98362025-04-30T06:46:26.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d36f35ee-2f4b-47ab-a4d0-65e2de30bc93", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98372025-04-30T06:46:26.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
98382025-04-30T06:46:26.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8086f44c-f08b-49c6-abce-1910b9eea606", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98392025-04-30T06:46:26.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
98402025-04-30T06:46:26.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df624e52-f84f-46e4-a2a4-1075d44e0039", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98412025-04-30T06:46:26.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
98422025-04-30T06:46:26.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3989fe62-b3bc-4457-b0ed-40e41561bdc2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98432025-04-30T06:46:26.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
98442025-04-30T06:46:26.270ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98452025-04-30T06:46:26.270ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98462025-04-30T06:46:26.270ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98472025-04-30T06:46:26.270ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
98482025-04-30T06:46:26.271ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98492025-04-30T06:46:26.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef8bb9bd-df33-40fb-90b6-ae01a713eb70", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98502025-04-30T06:46:26.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
98512025-04-30T06:46:26.274ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e82c635f-0ce9-484a-9ba3-b45eb35202cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98522025-04-30T06:46:26.274ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
98532025-04-30T06:46:26.274ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98542025-04-30T06:46:26.281ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98552025-04-30T06:46:26.282ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98562025-04-30T06:46:26.282ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
98572025-04-30T06:46:26.282ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98582025-04-30T06:46:26.286ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98592025-04-30T06:46:26.286ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
98602025-04-30T06:46:26.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98612025-04-30T06:46:26.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f61cfe2-9b6e-4e72-96fb-c588119d76f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98622025-04-30T06:46:26.288ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
98632025-04-30T06:46:26.289ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b32b7925-bdb3-4bf3-9615-3e2fc0d8c843", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98642025-04-30T06:46:26.289ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
98652025-04-30T06:46:26.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ae7ce22-87b0-4e31-812e-0c797d789848", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98662025-04-30T06:46:26.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
98672025-04-30T06:46:26.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "198b3437-ac73-4618-bffb-a706ab54a363", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98682025-04-30T06:46:26.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
98692025-04-30T06:46:26.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d129fbd-7e27-4d2e-9dea-733309d9d19f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:25 GMT"} })
98702025-04-30T06:46:26.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
98712025-04-30T06:46:26.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e27937ba-a278-461b-af2b-46d963ff9f79", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98722025-04-30T06:46:26.302ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
98732025-04-30T06:46:26.308ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79a90c9d-cce1-44ca-b541-3e4fe60b2986", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98742025-04-30T06:46:26.308ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
98752025-04-30T06:46:26.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a1ff86-c3a7-4fcd-a9db-b5e6e8c37910", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98762025-04-30T06:46:26.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
98772025-04-30T06:46:26.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "635db52e-7cb0-478a-92c5-c5e0565b8026", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98782025-04-30T06:46:26.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
98792025-04-30T06:46:26.318ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
98802025-04-30T06:46:26.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c22b1d-877e-45c2-9191-b43b9f2b74c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98812025-04-30T06:46:26.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
98822025-04-30T06:46:26.323ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "caad5257-4f8c-472f-92a7-381b0e718923", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98832025-04-30T06:46:26.323ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
98842025-04-30T06:46:26.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f47d72c-e26b-4f11-b897-3da364f283d2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98852025-04-30T06:46:26.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
98862025-04-30T06:46:26.327ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd50b1c5-72fc-4c95-abc2-c88700f48f0d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98872025-04-30T06:46:26.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
98882025-04-30T06:46:26.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c71747aa-2a9d-41c7-a4ba-7f011b7b5bc8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98892025-04-30T06:46:26.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
98902025-04-30T06:46:26.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17ffaba9-052b-4bfe-a970-d17e5267593f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98912025-04-30T06:46:26.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
98922025-04-30T06:46:26.335ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5063624-57ed-44c1-9b73-19227a9d6cdc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
98932025-04-30T06:46:26.335ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98942025-04-30T06:46:26.335ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98952025-04-30T06:46:26.335ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98962025-04-30T06:46:26.335ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98972025-04-30T06:46:26.336ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98982025-04-30T06:46:26.340ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98992025-04-30T06:46:26.344ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99002025-04-30T06:46:26.344ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99012025-04-30T06:46:26.344ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99022025-04-30T06:46:26.345ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99032025-04-30T06:46:26.345ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99042025-04-30T06:46:26.347ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99052025-04-30T06:46:26.351ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99062025-04-30T06:46:26.355ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99072025-04-30T06:46:26.355ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99082025-04-30T06:46:26.355ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99092025-04-30T06:46:26.356ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99102025-04-30T06:46:26.413ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
99112025-04-30T06:46:26.413ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
99122025-04-30T06:46:26.414ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99132025-04-30T06:46:26.414ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99142025-04-30T06:46:26.414ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99152025-04-30T06:46:26.414ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99162025-04-30T06:46:26.415ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99172025-04-30T06:46:26.415ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:60900 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99182025-04-30T06:46:26.417ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:60900 req_id = 0a91a2d4-d681-4c95-970c-1c1e87e5db3a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99192025-04-30T06:46:26.418ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1154 local_addr = [::1]:54590 method = GET remote_addr = [::1]:60900 req_id = 0a91a2d4-d681-4c95-970c-1c1e87e5db3a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99202025-04-30T06:46:26.420ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a91a2d4-d681-4c95-970c-1c1e87e5db3a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
99212025-04-30T06:46:26.424ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99222025-04-30T06:46:26.425ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99232025-04-30T06:46:26.425ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
99242025-04-30T06:46:26.434ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99252025-04-30T06:46:26.434ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99262025-04-30T06:46:26.434ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99272025-04-30T06:46:26.434ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99282025-04-30T06:46:26.435ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99292025-04-30T06:46:26.436ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99302025-04-30T06:46:26.443ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99312025-04-30T06:46:26.443ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99322025-04-30T06:46:26.443ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99332025-04-30T06:46:26.443ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99342025-04-30T06:46:26.451ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99352025-04-30T06:46:26.455ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99362025-04-30T06:46:26.455ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99372025-04-30T06:46:26.455ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99382025-04-30T06:46:26.456ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99392025-04-30T06:46:26.458ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99402025-04-30T06:46:26.458ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99412025-04-30T06:46:26.458ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99422025-04-30T06:46:26.517ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99432025-04-30T06:46:26.517ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99442025-04-30T06:46:26.518ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 414.078876ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99452025-04-30T06:46:26.518ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99462025-04-30T06:46:26.518ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99472025-04-30T06:46:26.518ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99482025-04-30T06:46:26.518ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99492025-04-30T06:46:26.518ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99502025-04-30T06:46:26.518ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99512025-04-30T06:46:26.521ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99522025-04-30T06:46:26.526ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99532025-04-30T06:46:26.526ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99542025-04-30T06:46:26.526ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99552025-04-30T06:46:26.528ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99562025-04-30T06:46:26.541ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
99572025-04-30T06:46:26.551ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
99582025-04-30T06:46:26.551ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99592025-04-30T06:46:26.553ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 51 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Q............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57274 req_id = f222b397-8e2b-4db2-b271-64f200221161
99602025-04-30T06:46:26.553ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20881,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57274 req_id = f222b397-8e2b-4db2-b271-64f200221161
99612025-04-30T06:46:26.553ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
99622025-04-30T06:46:26.553ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99632025-04-30T06:46:26.553ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57274 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f222b397-8e2b-4db2-b271-64f200221161
99642025-04-30T06:46:26.554ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4451, tv_nsec: 353035296 } })
99652025-04-30T06:46:26.554ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
99662025-04-30T06:46:26.554ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c5 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57441 req_id = 2d55218d-5ce5-44f0-9e9a-863d9a5e9adc
99672025-04-30T06:46:26.554ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57441 req_id = 2d55218d-5ce5-44f0-9e9a-863d9a5e9adc
99682025-04-30T06:46:26.555ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99692025-04-30T06:46:26.555ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57441 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2d55218d-5ce5-44f0-9e9a-863d9a5e9adc
99702025-04-30T06:46:26.613ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99712025-04-30T06:46:26.615ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99722025-04-30T06:46:26.615ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99732025-04-30T06:46:26.615ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99742025-04-30T06:46:26.615ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99752025-04-30T06:46:26.617ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99762025-04-30T06:46:26.617ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b7 fc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61509 req_id = 2cfcad97-11a7-4b4e-a1c7-055891063f71
99772025-04-30T06:46:26.617ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47100,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61509 req_id = 2cfcad97-11a7-4b4e-a1c7-055891063f71
99782025-04-30T06:46:26.618ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
99792025-04-30T06:46:26.619ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47100, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61509 req_id = 2cfcad97-11a7-4b4e-a1c7-055891063f71
99802025-04-30T06:46:26.620ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
99812025-04-30T06:46:26.620ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
99822025-04-30T06:46:26.620ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
99832025-04-30T06:46:26.621ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99842025-04-30T06:46:26.627ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99852025-04-30T06:46:26.627ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99862025-04-30T06:46:26.627ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99872025-04-30T06:46:26.627ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99882025-04-30T06:46:26.629ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8529950-b5d8-4588-8cb4-9efd6cb7907a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
99892025-04-30T06:46:26.630ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
99902025-04-30T06:46:26.630ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99912025-04-30T06:46:26.631ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
99922025-04-30T06:46:26.631ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99932025-04-30T06:46:26.632ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d7111bc-ec05-43b7-a580-e6a841db8c40", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
99942025-04-30T06:46:26.632ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
99952025-04-30T06:46:26.635ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99962025-04-30T06:46:26.635ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaa10126-6e78-4f62-8e93-bf205065b144", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
99972025-04-30T06:46:26.635ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
99982025-04-30T06:46:26.635ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99992025-04-30T06:46:26.638ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100002025-04-30T06:46:26.639ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100012025-04-30T06:46:26.640ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100022025-04-30T06:46:26.640ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100032025-04-30T06:46:26.642ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec51aea6-c99d-450f-8ccc-392fd31ad008", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100042025-04-30T06:46:26.642ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
100052025-04-30T06:46:26.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9828b094-0a5c-484a-917f-d426b66a171e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100062025-04-30T06:46:26.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
100072025-04-30T06:46:26.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1120cc64-3e39-49c1-be64-b118173a15ed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100082025-04-30T06:46:26.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
100092025-04-30T06:46:26.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39eb56d8-2e8f-4ba6-a599-6830749eee9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100102025-04-30T06:46:26.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
100112025-04-30T06:46:26.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aaca2cf-927f-483d-a970-2b718aec312d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100122025-04-30T06:46:26.656ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
100132025-04-30T06:46:26.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab0c82a5-19cd-4384-849d-adb03a6c4841", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100142025-04-30T06:46:26.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
100152025-04-30T06:46:26.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8e2a888-3d6b-4ef2-9c7c-fb3baae68e68", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100162025-04-30T06:46:26.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
100172025-04-30T06:46:26.665ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a281ce-5a7b-4565-874f-bd92727bedbb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100182025-04-30T06:46:26.665ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
100192025-04-30T06:46:26.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ca3434-ed9b-42a0-9b24-c8f6a5f43811", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100202025-04-30T06:46:26.669ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
100212025-04-30T06:46:26.673ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "778ad093-8034-4cb7-a502-0c4df8fa6358", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100222025-04-30T06:46:26.675ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
100232025-04-30T06:46:26.681ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51daaf46-f8be-47f3-932c-dda4f5c10824", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100242025-04-30T06:46:26.681ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
100252025-04-30T06:46:26.687ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
100262025-04-30T06:46:26.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "daaeb9a6-5853-4681-b87f-a5916948b058", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100272025-04-30T06:46:26.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
100282025-04-30T06:46:26.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fb0ef1a-0daa-4b6d-b1c0-b8f6c08a108b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100292025-04-30T06:46:26.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
100302025-04-30T06:46:26.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b4bdf7d-2905-4f8a-b372-e22392d9f553", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100312025-04-30T06:46:26.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
100322025-04-30T06:46:26.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e1b492-3d74-49b7-9413-12a9295db82a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100332025-04-30T06:46:26.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
100342025-04-30T06:46:26.700ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcf96aa4-5f28-41d1-ab36-a65c650c6294", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100352025-04-30T06:46:26.700ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
100362025-04-30T06:46:26.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "648d3c54-77ba-4664-8f6e-c5c7fdae494c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100372025-04-30T06:46:26.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
100382025-04-30T06:46:26.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de2780a-7dd6-4429-bb72-7f594b663caa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100392025-04-30T06:46:26.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
100402025-04-30T06:46:26.707ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd0320b9-c276-4946-95a3-e9ca82137ed8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100412025-04-30T06:46:26.707ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
100422025-04-30T06:46:26.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6be5c7-434f-455c-80f1-dbfc58181996", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100432025-04-30T06:46:26.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
100442025-04-30T06:46:26.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b9af1f-b163-49bb-81d8-2a8f0d9f9f93", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100452025-04-30T06:46:26.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
100462025-04-30T06:46:26.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13f00b5-bcf9-4883-9106-c1ba489ed319", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100472025-04-30T06:46:26.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
100482025-04-30T06:46:26.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff5cb226-b443-41ea-a155-41c5cabb1b73", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100492025-04-30T06:46:26.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
100502025-04-30T06:46:26.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5faf68fa-bc9d-4b85-8fc8-586ac937150b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100512025-04-30T06:46:26.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
100522025-04-30T06:46:26.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f563d2a-3b1b-441e-8d75-6ba7f032aca8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100532025-04-30T06:46:26.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
100542025-04-30T06:46:26.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4feccf3d-7494-4781-b65a-9a967f5543ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100552025-04-30T06:46:26.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
100562025-04-30T06:46:26.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8799930d-b268-4395-b1e5-1c6779d4c401", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100572025-04-30T06:46:26.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
100582025-04-30T06:46:26.726ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ed33a38-e4a0-4429-a15d-971fbd266500", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100592025-04-30T06:46:26.726ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
100602025-04-30T06:46:26.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1775f85-16dc-49ee-bdb8-243c7ff50176", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100612025-04-30T06:46:26.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
100622025-04-30T06:46:26.730ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "033c037a-e6cd-4264-b975-b9a34b9294bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100632025-04-30T06:46:26.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
100642025-04-30T06:46:26.734ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7b4e6d6-72ef-4b66-aace-1e0a8f7c510c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100652025-04-30T06:46:26.734ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
100662025-04-30T06:46:26.736ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34897fb0-247f-482f-baac-64341ee6b78c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100672025-04-30T06:46:26.736ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
100682025-04-30T06:46:26.738ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4448817-05df-4ed7-9a0b-ed0a4d09421c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100692025-04-30T06:46:26.738ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
100702025-04-30T06:46:26.738ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100712025-04-30T06:46:26.738ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100722025-04-30T06:46:26.738ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100732025-04-30T06:46:26.738ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
100742025-04-30T06:46:26.739ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100752025-04-30T06:46:26.742ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100762025-04-30T06:46:26.765ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100772025-04-30T06:46:26.765ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100782025-04-30T06:46:26.765ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
100792025-04-30T06:46:26.767ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100802025-04-30T06:46:26.767ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5736ac1c-5b71-4a84-b57f-2e99829acda3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100812025-04-30T06:46:26.768ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
100822025-04-30T06:46:26.771ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100832025-04-30T06:46:26.771ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
100842025-04-30T06:46:26.773ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100852025-04-30T06:46:26.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9956770-77b5-4fc2-a171-8b0f16d366c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100862025-04-30T06:46:26.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
100872025-04-30T06:46:26.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a18d10e7-ab95-4a1b-b0f7-492948bc4878", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100882025-04-30T06:46:26.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
100892025-04-30T06:46:26.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f8b48ff-13cb-4e1f-b1ec-a5877219291d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100902025-04-30T06:46:26.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
100912025-04-30T06:46:26.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "974069b0-7c7b-4140-af20-53a58df3b441", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100922025-04-30T06:46:26.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
100932025-04-30T06:46:26.805ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f274515a-6dfc-4ad4-acdc-cf1be54e162c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100942025-04-30T06:46:26.805ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
100952025-04-30T06:46:26.808ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5492fb-4a2f-46e9-acbb-79a0e016a03e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100962025-04-30T06:46:26.808ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
100972025-04-30T06:46:26.809ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6468be8a-37bc-43c9-bd2d-a567901a8ceb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
100982025-04-30T06:46:26.809ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
100992025-04-30T06:46:26.811ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2014861-1a60-45a5-b412-b4c224fc6e85", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
101002025-04-30T06:46:26.811ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101012025-04-30T06:46:26.811ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101022025-04-30T06:46:26.811ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101032025-04-30T06:46:26.811ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101042025-04-30T06:46:26.812ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101052025-04-30T06:46:26.817ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
101062025-04-30T06:46:26.818ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101072025-04-30T06:46:26.823ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101082025-04-30T06:46:26.823ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101092025-04-30T06:46:26.823ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101102025-04-30T06:46:26.823ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101112025-04-30T06:46:26.823ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101122025-04-30T06:46:26.823ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101132025-04-30T06:46:26.826ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101142025-04-30T06:46:26.830ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101152025-04-30T06:46:26.830ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101162025-04-30T06:46:26.830ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101172025-04-30T06:46:26.831ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101182025-04-30T06:46:26.905ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
101192025-04-30T06:46:26.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
101202025-04-30T06:46:26.905ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101212025-04-30T06:46:26.905ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101222025-04-30T06:46:26.905ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101232025-04-30T06:46:26.906ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
101242025-04-30T06:46:26.906ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101252025-04-30T06:46:26.906ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:65040 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101262025-04-30T06:46:26.907ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:65040 req_id = e0c3ba17-766d-4dd8-af60-b39bdfd30b5e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101272025-04-30T06:46:26.908ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 442 local_addr = [::1]:54590 method = GET remote_addr = [::1]:65040 req_id = e0c3ba17-766d-4dd8-af60-b39bdfd30b5e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101282025-04-30T06:46:26.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0c3ba17-766d-4dd8-af60-b39bdfd30b5e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
101292025-04-30T06:46:26.909ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101302025-04-30T06:46:26.909ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101312025-04-30T06:46:26.910ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
101322025-04-30T06:46:26.912ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101332025-04-30T06:46:26.912ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101342025-04-30T06:46:26.912ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101352025-04-30T06:46:26.912ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101362025-04-30T06:46:26.912ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101372025-04-30T06:46:26.913ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101382025-04-30T06:46:26.919ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101392025-04-30T06:46:26.919ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101402025-04-30T06:46:26.919ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
101412025-04-30T06:46:26.919ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101422025-04-30T06:46:26.922ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101432025-04-30T06:46:26.929ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101442025-04-30T06:46:26.929ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101452025-04-30T06:46:26.929ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101462025-04-30T06:46:26.931ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101472025-04-30T06:46:26.933ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101482025-04-30T06:46:26.933ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
101492025-04-30T06:46:26.934ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101502025-04-30T06:46:26.976ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101512025-04-30T06:46:26.976ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101522025-04-30T06:46:26.976ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 458.844728ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101532025-04-30T06:46:26.977ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101542025-04-30T06:46:26.977ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101552025-04-30T06:46:26.977ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101562025-04-30T06:46:26.977ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101572025-04-30T06:46:26.977ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101582025-04-30T06:46:26.977ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101592025-04-30T06:46:26.979ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101602025-04-30T06:46:26.979ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101612025-04-30T06:46:26.979ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
101622025-04-30T06:46:26.981ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
101632025-04-30T06:46:26.981ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101642025-04-30T06:46:26.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101652025-04-30T06:46:26.985ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101662025-04-30T06:46:26.985ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101672025-04-30T06:46:26.986ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101682025-04-30T06:46:26.992ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
101692025-04-30T06:46:26.993ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
101702025-04-30T06:46:26.993ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
101712025-04-30T06:46:26.993ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
101722025-04-30T06:46:26.993ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101732025-04-30T06:46:26.995ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 94 46 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .F..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61175 req_id = cfccde45-ced7-4d81-babf-45034f9be531
101742025-04-30T06:46:26.995ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37958,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61175 req_id = cfccde45-ced7-4d81-babf-45034f9be531
101752025-04-30T06:46:26.995ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
101762025-04-30T06:46:26.996ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101772025-04-30T06:46:26.996ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61175 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cfccde45-ced7-4d81-babf-45034f9be531
101782025-04-30T06:46:26.996ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4451, tv_nsec: 795515789 } })
101792025-04-30T06:46:26.996ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
101802025-04-30T06:46:26.999ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
101812025-04-30T06:46:26.999ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101822025-04-30T06:46:26.999ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 22 04 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e "...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59896 req_id = ae09820e-f707-4354-84b4-f6d08c72cbba
101832025-04-30T06:46:26.999ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59896 req_id = ae09820e-f707-4354-84b4-f6d08c72cbba
101842025-04-30T06:46:26.999ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101852025-04-30T06:46:26.999ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59896 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ae09820e-f707-4354-84b4-f6d08c72cbba
101862025-04-30T06:46:27.073ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101872025-04-30T06:46:27.073ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101882025-04-30T06:46:27.073ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101892025-04-30T06:46:27.073ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101902025-04-30T06:46:27.074ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
101912025-04-30T06:46:27.074ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101922025-04-30T06:46:27.075ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4b a7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c K............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61303 req_id = 8ebecd41-1fc1-446f-a4e2-cb84af354340
101932025-04-30T06:46:27.075ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19367,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61303 req_id = 8ebecd41-1fc1-446f-a4e2-cb84af354340
101942025-04-30T06:46:27.075ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
101952025-04-30T06:46:27.075ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19367, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61303 req_id = 8ebecd41-1fc1-446f-a4e2-cb84af354340
101962025-04-30T06:46:27.076ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
101972025-04-30T06:46:27.076ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
101982025-04-30T06:46:27.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
101992025-04-30T06:46:27.077ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102002025-04-30T06:46:27.085ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102012025-04-30T06:46:27.085ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102022025-04-30T06:46:27.085ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
102032025-04-30T06:46:27.086ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102042025-04-30T06:46:27.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "91113595-6a7f-4d38-8134-a213ab745400", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102052025-04-30T06:46:27.089ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
102062025-04-30T06:46:27.089ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
102072025-04-30T06:46:27.099ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102082025-04-30T06:46:27.099ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
102092025-04-30T06:46:27.100ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102102025-04-30T06:46:27.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4d1e4d3-153b-4bcf-94e2-ea10ee3e2fe5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102112025-04-30T06:46:27.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
102122025-04-30T06:46:27.122ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd27775e-8477-46f2-8022-71a514c07c3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102132025-04-30T06:46:27.122ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
102142025-04-30T06:46:27.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a5ba26-641d-4c1c-b9ea-c20abd6a7b0c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102152025-04-30T06:46:27.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
102162025-04-30T06:46:27.130ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c451b20-3dc1-4100-b040-655c56f98c17", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102172025-04-30T06:46:27.130ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
102182025-04-30T06:46:27.133ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cad037d3-14cf-45df-8d2c-ca3b97517e01", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102192025-04-30T06:46:27.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
102202025-04-30T06:46:27.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c14d3a9-dfb6-453e-b1a7-4e72d1432fab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102212025-04-30T06:46:27.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
102222025-04-30T06:46:27.140ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b78ff0b2-37a1-46f3-9a33-0485a8d9cfc3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102232025-04-30T06:46:27.140ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
102242025-04-30T06:46:27.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16551621-f239-40a5-9173-bff2bcdadf1d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102252025-04-30T06:46:27.142ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
102262025-04-30T06:46:27.143ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4c179a5-8594-4285-b738-fa0ba404c4b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102272025-04-30T06:46:27.143ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
102282025-04-30T06:46:27.145ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26abeba-34af-4879-9cad-6127d0bb2875", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102292025-04-30T06:46:27.147ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
102302025-04-30T06:46:27.148ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "299dd6e3-6bbe-4a5c-93d0-bdd5fc84e63c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102312025-04-30T06:46:27.149ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
102322025-04-30T06:46:27.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "732646b7-f769-4fc2-b1fc-44279d87517a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102332025-04-30T06:46:27.151ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
102342025-04-30T06:46:27.153ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5f76ca5-2372-4944-948a-afa03a9e910a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102352025-04-30T06:46:27.153ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
102362025-04-30T06:46:27.156ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c180cf1-256b-43b9-a0fc-e1649188a4f4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102372025-04-30T06:46:27.156ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
102382025-04-30T06:46:27.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c69a8df-6463-4796-8946-af11ba735f87", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102392025-04-30T06:46:27.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
102402025-04-30T06:46:27.160ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
102412025-04-30T06:46:27.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f2e21d-b292-41ba-96c0-c56a3ae26a4a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102422025-04-30T06:46:27.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
102432025-04-30T06:46:27.166ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b150584f-827c-408c-87bd-fca6188c871d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102442025-04-30T06:46:27.166ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
102452025-04-30T06:46:27.168ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da977126-6351-4470-aefb-d701cf62a721", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102462025-04-30T06:46:27.168ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
102472025-04-30T06:46:27.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a625fb3-37e1-4589-bdad-189d7addc5ad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102482025-04-30T06:46:27.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
102492025-04-30T06:46:27.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37311032-1119-4788-9842-36504a23c2dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102502025-04-30T06:46:27.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
102512025-04-30T06:46:27.173ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f56c24-ae60-4efa-a6ac-418407ebc085", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102522025-04-30T06:46:27.173ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
102532025-04-30T06:46:27.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b93d119-da8f-4727-ae68-7c00964531a4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102542025-04-30T06:46:27.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
102552025-04-30T06:46:27.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40c82685-4b00-4296-920f-d4533ac17e96", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102562025-04-30T06:46:27.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
102572025-04-30T06:46:27.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca20644e-da61-4942-88d1-0b7a3a034e0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102582025-04-30T06:46:27.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
102592025-04-30T06:46:27.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe6a7bc3-bb0e-4d57-9a5e-28f7e709053a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102602025-04-30T06:46:27.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
102612025-04-30T06:46:27.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c76aae0-0772-4ba0-860b-b84afa3eaa11", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102622025-04-30T06:46:27.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
102632025-04-30T06:46:27.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25f554bb-bd88-4572-ae85-cd0bb5765ff7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102642025-04-30T06:46:27.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
102652025-04-30T06:46:27.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3304f78-0589-418e-bdd1-e885d05933fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102662025-04-30T06:46:27.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
102672025-04-30T06:46:27.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee1e6e6d-46ae-491b-8e60-6cf0b574efcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102682025-04-30T06:46:27.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
102692025-04-30T06:46:27.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a91691ec-8a2c-4740-bd7a-3ff3f63a7063", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102702025-04-30T06:46:27.195ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
102712025-04-30T06:46:27.196ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9909953-fa01-4737-974a-af1cada5c118", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102722025-04-30T06:46:27.196ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
102732025-04-30T06:46:27.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd612150-91dd-4023-ba7d-9424d036380e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102742025-04-30T06:46:27.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
102752025-04-30T06:46:27.199ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c01b8f25-a92b-44d9-9d92-80c04db1b6e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102762025-04-30T06:46:27.200ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
102772025-04-30T06:46:27.202ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9d2eee2-dd9c-4914-98eb-c575ea88617e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102782025-04-30T06:46:27.202ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
102792025-04-30T06:46:27.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3944e126-8abe-4fcc-9eb5-d5efeede7d71", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102802025-04-30T06:46:27.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
102812025-04-30T06:46:27.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3f9665f-986a-455b-8f7d-583ec60f138c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102822025-04-30T06:46:27.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
102832025-04-30T06:46:27.211ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102842025-04-30T06:46:27.211ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102852025-04-30T06:46:27.211ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102862025-04-30T06:46:27.211ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
102872025-04-30T06:46:27.212ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102882025-04-30T06:46:27.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4149e0e7-1beb-4db5-a2b4-9596d5257504", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102892025-04-30T06:46:27.213ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
102902025-04-30T06:46:27.214ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102912025-04-30T06:46:27.220ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102922025-04-30T06:46:27.220ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102932025-04-30T06:46:27.220ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
102942025-04-30T06:46:27.220ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102952025-04-30T06:46:27.221ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e2c1192-75ca-4eca-a7b2-b0e8985bf7c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
102962025-04-30T06:46:27.222ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
102972025-04-30T06:46:27.223ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102982025-04-30T06:46:27.223ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
102992025-04-30T06:46:27.224ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103002025-04-30T06:46:27.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1039e3-fb90-49dd-873c-d09990c8d7cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103012025-04-30T06:46:27.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
103022025-04-30T06:46:27.227ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "392eca7b-0c1f-4674-b6c8-62206c34df70", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103032025-04-30T06:46:27.227ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
103042025-04-30T06:46:27.228ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a72898d2-4337-4051-9aa4-0e8f237ebbeb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103052025-04-30T06:46:27.229ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
103062025-04-30T06:46:27.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baad5087-6c9a-42fe-a432-ebd055588a01", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103072025-04-30T06:46:27.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
103082025-04-30T06:46:27.231ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22ef43f8-88b6-4c83-a095-fc535eb5a832", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103092025-04-30T06:46:27.231ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
103102025-04-30T06:46:27.232ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bfeb7ab-7a8b-400f-b034-7861ed4c1b5d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:26 GMT"} })
103112025-04-30T06:46:27.232ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103122025-04-30T06:46:27.232ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103132025-04-30T06:46:27.232ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103142025-04-30T06:46:27.232ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103152025-04-30T06:46:27.233ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103162025-04-30T06:46:27.238ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103172025-04-30T06:46:27.244ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103182025-04-30T06:46:27.244ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103192025-04-30T06:46:27.244ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103202025-04-30T06:46:27.244ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103212025-04-30T06:46:27.244ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103222025-04-30T06:46:27.244ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103232025-04-30T06:46:27.247ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103242025-04-30T06:46:27.251ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103252025-04-30T06:46:27.251ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103262025-04-30T06:46:27.251ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103272025-04-30T06:46:27.252ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103282025-04-30T06:46:27.304ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
103292025-04-30T06:46:27.304ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
103302025-04-30T06:46:27.305ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:37011 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103312025-04-30T06:46:27.306ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:37011 req_id = bb7604cc-6584-416d-983b-3307818bb73a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103322025-04-30T06:46:27.306ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 387 local_addr = [::1]:54590 method = GET remote_addr = [::1]:37011 req_id = bb7604cc-6584-416d-983b-3307818bb73a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103332025-04-30T06:46:27.306ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb7604cc-6584-416d-983b-3307818bb73a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
103342025-04-30T06:46:27.308ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103352025-04-30T06:46:27.308ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103362025-04-30T06:46:27.309ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
103372025-04-30T06:46:27.311ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103382025-04-30T06:46:27.311ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103392025-04-30T06:46:27.311ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103402025-04-30T06:46:27.311ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103412025-04-30T06:46:27.311ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103422025-04-30T06:46:27.315ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103432025-04-30T06:46:27.321ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103442025-04-30T06:46:27.321ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103452025-04-30T06:46:27.321ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103462025-04-30T06:46:27.322ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103472025-04-30T06:46:27.326ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
103482025-04-30T06:46:27.360ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
103492025-04-30T06:46:27.361ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
103502025-04-30T06:46:27.361ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 384.637054ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
103512025-04-30T06:46:27.361ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
103522025-04-30T06:46:27.361ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103532025-04-30T06:46:27.361ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103542025-04-30T06:46:27.361ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103552025-04-30T06:46:27.361ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103562025-04-30T06:46:27.362ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103572025-04-30T06:46:27.367ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103582025-04-30T06:46:27.375ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103592025-04-30T06:46:27.375ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103602025-04-30T06:46:27.375ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103612025-04-30T06:46:27.376ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103622025-04-30T06:46:27.377ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103632025-04-30T06:46:27.378ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103642025-04-30T06:46:27.378ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103652025-04-30T06:46:27.378ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
103662025-04-30T06:46:27.379ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103672025-04-30T06:46:27.381ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103682025-04-30T06:46:27.391ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103692025-04-30T06:46:27.391ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103702025-04-30T06:46:27.391ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
103712025-04-30T06:46:27.391ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103722025-04-30T06:46:27.391ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
103732025-04-30T06:46:27.391ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
103742025-04-30T06:46:27.393ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: de 37 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .7..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61512 req_id = ac641298-7f74-4a7f-a5ea-c02aca28cfca
103752025-04-30T06:46:27.393ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56887,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61512 req_id = ac641298-7f74-4a7f-a5ea-c02aca28cfca
103762025-04-30T06:46:27.393ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
103772025-04-30T06:46:27.393ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103782025-04-30T06:46:27.393ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61512 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ac641298-7f74-4a7f-a5ea-c02aca28cfca
103792025-04-30T06:46:27.393ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4452, tv_nsec: 192782177 } })
103802025-04-30T06:46:27.394ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
103812025-04-30T06:46:27.394ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2e 50 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .P..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58971 req_id = d0cf4ef8-75cf-4fa4-b9c0-41a5e8dac73a
103822025-04-30T06:46:27.394ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11856,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58971 req_id = d0cf4ef8-75cf-4fa4-b9c0-41a5e8dac73a
103832025-04-30T06:46:27.394ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103842025-04-30T06:46:27.394ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58971 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d0cf4ef8-75cf-4fa4-b9c0-41a5e8dac73a
103852025-04-30T06:46:27.453ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
103862025-04-30T06:46:27.454ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103872025-04-30T06:46:27.454ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
103882025-04-30T06:46:27.455ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103892025-04-30T06:46:27.456ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5d b2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ]............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62334 req_id = f99f06b6-a884-4dbc-9235-b9cb81e64c72
103902025-04-30T06:46:27.456ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23986,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62334 req_id = f99f06b6-a884-4dbc-9235-b9cb81e64c72
103912025-04-30T06:46:27.457ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
103922025-04-30T06:46:27.457ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23986, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62334 req_id = f99f06b6-a884-4dbc-9235-b9cb81e64c72
103932025-04-30T06:46:27.458ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
103942025-04-30T06:46:27.458ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
103952025-04-30T06:46:27.458ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
103962025-04-30T06:46:27.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0bf9570-7321-460f-a81c-d3392cdbfc74", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
103972025-04-30T06:46:27.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
103982025-04-30T06:46:27.483ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5567f9d8-a000-4e5c-bfe6-3c9e5bc9b209", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
103992025-04-30T06:46:27.483ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
104002025-04-30T06:46:27.493ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e19520cd-f698-4f65-bb37-ac9abf162f6e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104012025-04-30T06:46:27.493ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
104022025-04-30T06:46:27.496ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ff6e1e-76ce-48c0-82c5-45d46d64ebc2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104032025-04-30T06:46:27.497ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
104042025-04-30T06:46:27.499ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e2825ff-ace6-42e9-aea4-472a2744aff0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104052025-04-30T06:46:27.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
104062025-04-30T06:46:27.504ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c79d6adc-7907-4213-8b11-96f2648ba671", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104072025-04-30T06:46:27.505ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
104082025-04-30T06:46:27.505ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
104092025-04-30T06:46:27.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "601e03c1-2f8a-47d6-8386-e115826808be", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104102025-04-30T06:46:27.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
104112025-04-30T06:46:27.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74fa2ba5-1314-41a9-a370-6d073f73d820", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104122025-04-30T06:46:27.510ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
104132025-04-30T06:46:27.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4eebdcb-99d4-494a-be58-bc3199b416dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104142025-04-30T06:46:27.512ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
104152025-04-30T06:46:27.513ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "309dd712-47ff-4a8f-a6e0-370b49590830", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104162025-04-30T06:46:27.514ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
104172025-04-30T06:46:27.515ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd01573d-649c-4916-a04f-95d01f59077b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104182025-04-30T06:46:27.516ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
104192025-04-30T06:46:27.517ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b91e003-07e3-4ce5-a941-7763bffc6f92", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104202025-04-30T06:46:27.517ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
104212025-04-30T06:46:27.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2299c177-b232-40c0-8504-255c206ff53c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104222025-04-30T06:46:27.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
104232025-04-30T06:46:27.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "388a2475-215e-4b05-85e9-14b18f01fa59", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104242025-04-30T06:46:27.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
104252025-04-30T06:46:27.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c7ee31-a4c8-455a-bf0d-eba426dab154", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104262025-04-30T06:46:27.523ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
104272025-04-30T06:46:27.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "842752fd-f657-4ea5-843a-2083d3f7f234", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104282025-04-30T06:46:27.526ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
104292025-04-30T06:46:27.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38379a2c-08b2-4a2f-82b4-aa523e903b8f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104302025-04-30T06:46:27.529ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
104312025-04-30T06:46:27.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d72413-5d76-4b55-a479-9fa26fc87900", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104322025-04-30T06:46:27.531ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
104332025-04-30T06:46:27.535ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e464468a-7a05-46f2-acfd-0d9c11579f6b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104342025-04-30T06:46:27.535ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
104352025-04-30T06:46:27.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffe10cb2-822d-49cd-864d-2c51f4962764", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104362025-04-30T06:46:27.536ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
104372025-04-30T06:46:27.537ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9746c9ae-84ac-465f-9749-a3bf0dfa7d46", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104382025-04-30T06:46:27.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
104392025-04-30T06:46:27.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af0279e3-79b8-4007-aaaf-72b44859e3b1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104402025-04-30T06:46:27.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
104412025-04-30T06:46:27.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7605f8-200e-46b6-869f-dbaa79994863", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104422025-04-30T06:46:27.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
104432025-04-30T06:46:27.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "630f1c1a-cca3-4308-90ae-7e02128d47da", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104442025-04-30T06:46:27.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
104452025-04-30T06:46:27.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd04be2-39f8-4474-ae97-f4e4d2dd652a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104462025-04-30T06:46:27.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
104472025-04-30T06:46:27.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b89f81-906d-4c7d-af03-47de34459274", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104482025-04-30T06:46:27.546ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
104492025-04-30T06:46:27.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e79a9bb-0a08-43dd-aaf5-0ba66554c6d7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104502025-04-30T06:46:27.548ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
104512025-04-30T06:46:27.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
104522025-04-30T06:46:27.549ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104532025-04-30T06:46:27.550ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 300f7f0d-e509-4265-bd5e-efbe7bdc18a1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104542025-04-30T06:46:27.551ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 300f7f0d-e509-4265-bd5e-efbe7bdc18a1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104552025-04-30T06:46:27.551ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 300f7f0d-e509-4265-bd5e-efbe7bdc18a1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104562025-04-30T06:46:27.551ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 300f7f0d-e509-4265-bd5e-efbe7bdc18a1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104572025-04-30T06:46:27.555ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 5658 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 300f7f0d-e509-4265-bd5e-efbe7bdc18a1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
104582025-04-30T06:46:27.556ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "300f7f0d-e509-4265-bd5e-efbe7bdc18a1", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104592025-04-30T06:46:27.556ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104602025-04-30T06:46:27.556ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104612025-04-30T06:46:27.556ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104622025-04-30T06:46:27.556ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104632025-04-30T06:46:27.556ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
104642025-04-30T06:46:27.557ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104652025-04-30T06:46:27.557ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9dd039d3-bf05-4435-a747-9083b370eb49 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104662025-04-30T06:46:27.557ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9dd039d3-bf05-4435-a747-9083b370eb49 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104672025-04-30T06:46:27.557ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9dd039d3-bf05-4435-a747-9083b370eb49 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104682025-04-30T06:46:27.558ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9dd039d3-bf05-4435-a747-9083b370eb49 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104692025-04-30T06:46:27.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b150c9d-afcd-4f1b-ab3a-8c117d8ed2ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104702025-04-30T06:46:27.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
104712025-04-30T06:46:27.562ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104722025-04-30T06:46:27.568ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104732025-04-30T06:46:27.568ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104742025-04-30T06:46:27.568ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
104752025-04-30T06:46:27.568ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104762025-04-30T06:46:27.569ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 11943 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 9dd039d3-bf05-4435-a747-9083b370eb49 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
104772025-04-30T06:46:27.569ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd039d3-bf05-4435-a747-9083b370eb49", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104782025-04-30T06:46:27.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
104792025-04-30T06:46:27.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104802025-04-30T06:46:27.569ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
104812025-04-30T06:46:27.571ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104822025-04-30T06:46:27.571ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
104832025-04-30T06:46:27.572ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104842025-04-30T06:46:27.573ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b67118c-df79-4b7e-aa72-c7b412c736f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104852025-04-30T06:46:27.573ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
104862025-04-30T06:46:27.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f901c4f-3195-4c17-a047-0c94b2c8aa44", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104872025-04-30T06:46:27.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
104882025-04-30T06:46:27.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c955451-85a6-4b5c-b066-f513c2b28e7b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104892025-04-30T06:46:27.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
104902025-04-30T06:46:27.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf904a71-d92f-4632-b33f-75db29f190a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104912025-04-30T06:46:27.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
104922025-04-30T06:46:27.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69bc944-73fd-4578-be3c-256b2a95049c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104932025-04-30T06:46:27.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
104942025-04-30T06:46:27.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3de65a9-f2d3-47bd-aa1a-1b1c55da84a4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104952025-04-30T06:46:27.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
104962025-04-30T06:46:27.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a7a0ca1-e36e-43da-95f0-2eaf6af62b85", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104972025-04-30T06:46:27.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
104982025-04-30T06:46:27.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63f9cab9-8f10-47ea-9ac6-276e0acb6585", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
104992025-04-30T06:46:27.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
105002025-04-30T06:46:27.593ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a33f5695-efd5-4262-b829-ef5e97f40c8c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105012025-04-30T06:46:27.593ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
105022025-04-30T06:46:27.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f430afc-4461-4687-bfc5-8244e34a8a9a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105032025-04-30T06:46:27.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
105042025-04-30T06:46:27.602ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
105052025-04-30T06:46:27.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4575d28a-bd76-409b-b7a2-1f23c1e53c87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105062025-04-30T06:46:27.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
105072025-04-30T06:46:27.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41824d4d-709d-433b-8618-3be1e7390471", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105082025-04-30T06:46:27.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
105092025-04-30T06:46:27.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4016a5e2-c0b3-4907-a78e-8271799dd89f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105102025-04-30T06:46:27.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
105112025-04-30T06:46:27.608ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59b0a781-3e1c-4439-8ad2-b3d2c0c262c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105122025-04-30T06:46:27.608ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
105132025-04-30T06:46:27.609ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e2c768-248e-4da1-9adb-b3074d50056f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105142025-04-30T06:46:27.609ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
105152025-04-30T06:46:27.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "568275fa-d1db-4b49-86a1-4dedccfc6059", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105162025-04-30T06:46:27.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
105172025-04-30T06:46:27.611ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bdbeee3-d4e6-4cc0-8a1a-397280940e5b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105182025-04-30T06:46:27.611ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105192025-04-30T06:46:27.612ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105202025-04-30T06:46:27.612ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105212025-04-30T06:46:27.612ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105222025-04-30T06:46:27.612ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105232025-04-30T06:46:27.624ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105242025-04-30T06:46:27.629ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105252025-04-30T06:46:27.629ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105262025-04-30T06:46:27.629ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105272025-04-30T06:46:27.629ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105282025-04-30T06:46:27.629ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105292025-04-30T06:46:27.630ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105302025-04-30T06:46:27.640ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105312025-04-30T06:46:27.640ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105322025-04-30T06:46:27.647ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105332025-04-30T06:46:27.653ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105342025-04-30T06:46:27.653ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105352025-04-30T06:46:27.653ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105362025-04-30T06:46:27.653ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105372025-04-30T06:46:27.654ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105382025-04-30T06:46:27.654ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105392025-04-30T06:46:27.654ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105402025-04-30T06:46:27.654ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
105412025-04-30T06:46:27.654ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105422025-04-30T06:46:27.657ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105432025-04-30T06:46:27.657ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105442025-04-30T06:46:27.657ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105452025-04-30T06:46:27.657ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105462025-04-30T06:46:27.671ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105472025-04-30T06:46:27.673ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105482025-04-30T06:46:27.673ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105492025-04-30T06:46:27.673ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
105502025-04-30T06:46:27.674ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105512025-04-30T06:46:27.713ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
105522025-04-30T06:46:27.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
105532025-04-30T06:46:27.715ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:37243 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105542025-04-30T06:46:27.715ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:37243 req_id = 0811ef6f-674e-46de-b427-31a5091eae6c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105552025-04-30T06:46:27.715ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 366 local_addr = [::1]:54590 method = GET remote_addr = [::1]:37243 req_id = 0811ef6f-674e-46de-b427-31a5091eae6c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105562025-04-30T06:46:27.715ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105572025-04-30T06:46:27.715ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
105582025-04-30T06:46:27.716ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105592025-04-30T06:46:27.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0811ef6f-674e-46de-b427-31a5091eae6c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
105602025-04-30T06:46:27.718ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105612025-04-30T06:46:27.718ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105622025-04-30T06:46:27.718ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
105632025-04-30T06:46:27.719ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105642025-04-30T06:46:27.719ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105652025-04-30T06:46:27.719ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105662025-04-30T06:46:27.719ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105672025-04-30T06:46:27.720ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105682025-04-30T06:46:27.725ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105692025-04-30T06:46:27.729ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105702025-04-30T06:46:27.730ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105712025-04-30T06:46:27.730ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105722025-04-30T06:46:27.730ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105732025-04-30T06:46:27.777ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
105742025-04-30T06:46:27.784ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
105752025-04-30T06:46:27.787ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
105762025-04-30T06:46:27.788ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 425.785188ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
105772025-04-30T06:46:27.788ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
105782025-04-30T06:46:27.788ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105792025-04-30T06:46:27.788ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105802025-04-30T06:46:27.788ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105812025-04-30T06:46:27.788ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105822025-04-30T06:46:27.789ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105832025-04-30T06:46:27.792ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105842025-04-30T06:46:27.799ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105852025-04-30T06:46:27.799ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105862025-04-30T06:46:27.799ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105872025-04-30T06:46:27.799ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105882025-04-30T06:46:27.804ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
105892025-04-30T06:46:27.804ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
105902025-04-30T06:46:27.806ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 71 08 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 q............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64955 req_id = 229d6a84-c20e-484e-95c1-c9c28813d667
105912025-04-30T06:46:27.806ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28936,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64955 req_id = 229d6a84-c20e-484e-95c1-c9c28813d667
105922025-04-30T06:46:27.806ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
105932025-04-30T06:46:27.806ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
105942025-04-30T06:46:27.806ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64955 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 229d6a84-c20e-484e-95c1-c9c28813d667
105952025-04-30T06:46:27.807ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4452, tv_nsec: 606190755 } })
105962025-04-30T06:46:27.807ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
105972025-04-30T06:46:27.807ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a0 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .t..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61189 req_id = 9b2fa54f-4def-4276-8789-e29299b29a6d
105982025-04-30T06:46:27.807ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41076,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61189 req_id = 9b2fa54f-4def-4276-8789-e29299b29a6d
105992025-04-30T06:46:27.807ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106002025-04-30T06:46:27.807ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61189 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9b2fa54f-4def-4276-8789-e29299b29a6d
106012025-04-30T06:46:27.863ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106022025-04-30T06:46:27.863ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106032025-04-30T06:46:27.863ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106042025-04-30T06:46:27.863ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106052025-04-30T06:46:27.863ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106062025-04-30T06:46:27.864ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106072025-04-30T06:46:27.865ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0b 19 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54839 req_id = a84d1fbb-c8ea-4366-89a4-27d9176aefd2
106082025-04-30T06:46:27.865ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54839 req_id = a84d1fbb-c8ea-4366-89a4-27d9176aefd2
106092025-04-30T06:46:27.865ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
106102025-04-30T06:46:27.865ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2841, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54839 req_id = a84d1fbb-c8ea-4366-89a4-27d9176aefd2
106112025-04-30T06:46:27.865ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
106122025-04-30T06:46:27.865ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
106132025-04-30T06:46:27.865ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
106142025-04-30T06:46:27.868ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106152025-04-30T06:46:27.877ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106162025-04-30T06:46:27.877ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106172025-04-30T06:46:27.877ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106182025-04-30T06:46:27.878ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106192025-04-30T06:46:27.878ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d20156de-8b27-4ba0-8c5c-1e0dcaa6c1b0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106202025-04-30T06:46:27.878ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
106212025-04-30T06:46:27.927ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106222025-04-30T06:46:27.927ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106232025-04-30T06:46:27.928ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106242025-04-30T06:46:27.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a602ece0-6ed7-429b-bf29-442ee97c70d3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106252025-04-30T06:46:27.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
106262025-04-30T06:46:27.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81dfc061-afe4-4e31-bd38-946e034ec912", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106272025-04-30T06:46:27.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
106282025-04-30T06:46:27.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "492e6b62-d477-4c49-9224-8db5c2e9526e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106292025-04-30T06:46:27.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
106302025-04-30T06:46:27.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f19365e5-aed0-4bc7-8b4d-064906f0d6b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106312025-04-30T06:46:27.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
106322025-04-30T06:46:27.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bdcfcff-3608-4538-a9a9-ea9f0ca7c384", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106332025-04-30T06:46:27.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
106342025-04-30T06:46:27.944ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fea7cd0-ab8c-4b9d-b518-0748195b8b8a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106352025-04-30T06:46:27.944ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
106362025-04-30T06:46:27.948ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "269e3e41-ed8f-48e7-a64a-84f7406b0256", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106372025-04-30T06:46:27.948ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
106382025-04-30T06:46:27.950ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c22d660c-351a-4976-bf9f-ee996b0cdbd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106392025-04-30T06:46:27.950ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
106402025-04-30T06:46:27.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03445d7f-92dd-4852-a125-2568a11c29c6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106412025-04-30T06:46:27.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
106422025-04-30T06:46:27.954ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42efa25-21ff-4e3c-9831-480ae8cc0f97", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106432025-04-30T06:46:27.955ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
106442025-04-30T06:46:27.957ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bbf3ab7-de9e-4d16-bdc5-2996d5c39c97", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106452025-04-30T06:46:27.957ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
106462025-04-30T06:46:27.957ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
106472025-04-30T06:46:27.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82708811-e9d2-49bb-877c-dcbdfb18d572", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106482025-04-30T06:46:27.959ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
106492025-04-30T06:46:27.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "159b8312-50e7-41db-a937-6f54ed0fa340", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106502025-04-30T06:46:27.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
106512025-04-30T06:46:27.973ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "379f1cf0-e287-415c-a91f-2a11580703db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106522025-04-30T06:46:27.973ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
106532025-04-30T06:46:27.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be5821a-6146-4585-9cf8-6b9ac4415544", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106542025-04-30T06:46:27.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
106552025-04-30T06:46:27.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec59e5f-d0d7-4b91-8a7d-c8f3158e26f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106562025-04-30T06:46:27.978ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
106572025-04-30T06:46:27.979ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106582025-04-30T06:46:27.979ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106592025-04-30T06:46:27.979ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
106602025-04-30T06:46:27.980ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
106612025-04-30T06:46:27.981ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "965b230e-48ef-45b9-9dbf-5990a4551357", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106622025-04-30T06:46:27.981ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
106632025-04-30T06:46:27.989ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
106642025-04-30T06:46:27.989ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
106652025-04-30T06:46:27.993ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11431f71-fa03-4406-98c1-148317dc00bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106662025-04-30T06:46:27.993ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
106672025-04-30T06:46:27.994ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
106682025-04-30T06:46:27.994ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106692025-04-30T06:46:27.995ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1362001f-acf4-4ee8-986a-796ed31d6f84", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106702025-04-30T06:46:27.995ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
106712025-04-30T06:46:27.996ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9da12258-5507-451e-8541-ed530f6f850e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106722025-04-30T06:46:27.996ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
106732025-04-30T06:46:27.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "016c65cc-039e-4cee-9552-574706a87f8d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106742025-04-30T06:46:27.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
106752025-04-30T06:46:27.998ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c41f832d-128d-4e1e-9945-76f95ff35c78", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106762025-04-30T06:46:27.998ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
106772025-04-30T06:46:28.000ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d938df79-5b26-448b-90c0-6bfb2805d335", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106782025-04-30T06:46:28.000ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
106792025-04-30T06:46:28.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eca5b38f-d782-49a5-b323-cd5362af74ab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106802025-04-30T06:46:28.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
106812025-04-30T06:46:28.004ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b52b999-a143-4f0c-a9e2-4a666f2b9b16", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106822025-04-30T06:46:28.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
106832025-04-30T06:46:28.007ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7654eb4-3c04-4496-a9b8-5618f36525e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106842025-04-30T06:46:28.007ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
106852025-04-30T06:46:28.008ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106862025-04-30T06:46:28.008ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106872025-04-30T06:46:28.008ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106882025-04-30T06:46:28.008ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106892025-04-30T06:46:28.009ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106902025-04-30T06:46:28.011ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a735ac-2721-42f1-89c4-e8967ba5947b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
106912025-04-30T06:46:28.011ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
106922025-04-30T06:46:28.011ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106932025-04-30T06:46:28.018ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106942025-04-30T06:46:28.018ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106952025-04-30T06:46:28.018ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106962025-04-30T06:46:28.018ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106972025-04-30T06:46:28.028ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106982025-04-30T06:46:28.028ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
106992025-04-30T06:46:28.028ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107002025-04-30T06:46:28.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fee951ce-9d02-40e4-a7a8-46bcce61a3b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107012025-04-30T06:46:28.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
107022025-04-30T06:46:28.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c58f03b3-7edc-4078-b841-89b7ccdf3ad2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107032025-04-30T06:46:28.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
107042025-04-30T06:46:28.034ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5959fb-7cc0-4f05-bc08-b69ab5382b58", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107052025-04-30T06:46:28.034ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
107062025-04-30T06:46:28.035ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03befffe-87a8-40ce-82b1-45a61b11a31b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107072025-04-30T06:46:28.035ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
107082025-04-30T06:46:28.037ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89df9fcb-3827-4958-8b41-cc4ba4e6016a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107092025-04-30T06:46:28.037ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
107102025-04-30T06:46:28.042ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80d158fa-f36f-471a-adcc-8bc54c20774e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107112025-04-30T06:46:28.043ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
107122025-04-30T06:46:28.043ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9586c530-4488-4149-9f3e-97fed947a45c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107132025-04-30T06:46:28.043ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
107142025-04-30T06:46:28.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5cd78c3-a986-4e1b-8896-0ba869475f3a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107152025-04-30T06:46:28.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
107162025-04-30T06:46:28.052ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f2ec9c-cdad-4fd8-b86a-06cc289fef7b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107172025-04-30T06:46:28.052ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
107182025-04-30T06:46:28.055ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7c302e-7dd2-4ce1-a6fd-6b8a9da4f669", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107192025-04-30T06:46:28.055ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
107202025-04-30T06:46:28.059ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
107212025-04-30T06:46:28.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b4be7b-3818-4a79-9405-ff45349c6248", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107222025-04-30T06:46:28.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
107232025-04-30T06:46:28.066ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f8a7e6-3b77-487b-a286-f8832e26d8a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107242025-04-30T06:46:28.066ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
107252025-04-30T06:46:28.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91c0bb33-aedd-4689-a866-e3d57bf41dc4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107262025-04-30T06:46:28.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
107272025-04-30T06:46:28.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dae8bff-9829-47a1-9af6-3479dfcd968c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107282025-04-30T06:46:28.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
107292025-04-30T06:46:28.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfa93b38-7846-4c67-bb75-b9b7be8647df", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107302025-04-30T06:46:28.073ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
107312025-04-30T06:46:28.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d160ec2-e2a8-4707-af05-656f95955c9d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107322025-04-30T06:46:28.075ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
107332025-04-30T06:46:28.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa7d4e1-0e9c-4ba8-962c-e82332c7bf68", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
107342025-04-30T06:46:28.076ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107352025-04-30T06:46:28.076ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107362025-04-30T06:46:28.076ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107372025-04-30T06:46:28.076ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107382025-04-30T06:46:28.077ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107392025-04-30T06:46:28.086ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
107402025-04-30T06:46:28.089ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107412025-04-30T06:46:28.121ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107422025-04-30T06:46:28.121ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107432025-04-30T06:46:28.121ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107442025-04-30T06:46:28.121ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107452025-04-30T06:46:28.121ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107462025-04-30T06:46:28.121ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107472025-04-30T06:46:28.122ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107482025-04-30T06:46:28.122ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107492025-04-30T06:46:28.122ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107502025-04-30T06:46:28.123ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
107512025-04-30T06:46:28.124ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107522025-04-30T06:46:28.133ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107532025-04-30T06:46:28.149ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107542025-04-30T06:46:28.149ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107552025-04-30T06:46:28.149ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107562025-04-30T06:46:28.150ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107572025-04-30T06:46:28.151ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107582025-04-30T06:46:28.159ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107592025-04-30T06:46:28.159ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107602025-04-30T06:46:28.159ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
107612025-04-30T06:46:28.160ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107622025-04-30T06:46:28.241ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
107632025-04-30T06:46:28.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
107642025-04-30T06:46:28.242ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:62543 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107652025-04-30T06:46:28.242ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
107662025-04-30T06:46:28.242ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
107672025-04-30T06:46:28.242ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
107682025-04-30T06:46:28.243ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
107692025-04-30T06:46:28.243ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
107702025-04-30T06:46:28.243ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
107712025-04-30T06:46:28.243ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
107722025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
107732025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
107742025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
107752025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 27.274\u{b5}s iteration = 3 status = {"error":"task disabled"}
107762025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
107772025-04-30T06:46:28.244ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
107782025-04-30T06:46:28.244ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107792025-04-30T06:46:28.244ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
107802025-04-30T06:46:28.244ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
107812025-04-30T06:46:28.245ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
107822025-04-30T06:46:28.245ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
107832025-04-30T06:46:28.245ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
107842025-04-30T06:46:28.245ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107852025-04-30T06:46:28.245ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
107862025-04-30T06:46:28.245ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107872025-04-30T06:46:28.245ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107882025-04-30T06:46:28.246ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
107892025-04-30T06:46:28.246ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
107902025-04-30T06:46:28.246ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107912025-04-30T06:46:28.246ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107922025-04-30T06:46:28.248ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
107932025-04-30T06:46:28.248ZINFOtest_multiple_snapshots (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
107942025-04-30T06:46:28.248ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = instance_updater elapsed = 111.412\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
107952025-04-30T06:46:28.248ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
107962025-04-30T06:46:28.248ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
107972025-04-30T06:46:28.248ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107982025-04-30T06:46:28.249ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
107992025-04-30T06:46:28.249ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108002025-04-30T06:46:28.250ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108012025-04-30T06:46:28.250ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
108022025-04-30T06:46:28.250ZINFOtest_multiple_snapshots (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
108032025-04-30T06:46:28.250ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108042025-04-30T06:46:28.250ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108052025-04-30T06:46:28.250ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108062025-04-30T06:46:28.250ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108072025-04-30T06:46:28.253ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108082025-04-30T06:46:28.254ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108092025-04-30T06:46:28.254ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
108102025-04-30T06:46:28.255ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108112025-04-30T06:46:28.255ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:62543 req_id = 88c7cc4a-4833-4a9a-8fd4-891acff7fe62 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108122025-04-30T06:46:28.257ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1313 local_addr = [::1]:54590 method = GET remote_addr = [::1]:62543 req_id = 88c7cc4a-4833-4a9a-8fd4-891acff7fe62 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108132025-04-30T06:46:28.258ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "88c7cc4a-4833-4a9a-8fd4-891acff7fe62", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:27 GMT"} })
108142025-04-30T06:46:28.262ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108152025-04-30T06:46:28.262ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108162025-04-30T06:46:28.262ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
108172025-04-30T06:46:28.267ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108182025-04-30T06:46:28.267ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108192025-04-30T06:46:28.267ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108202025-04-30T06:46:28.268ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108212025-04-30T06:46:28.269ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108222025-04-30T06:46:28.270ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108232025-04-30T06:46:28.272ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108242025-04-30T06:46:28.273ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 25 a7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 %............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56424 req_id = 3e0ed7cc-8140-4b4c-b183-e75860b920da
108252025-04-30T06:46:28.274ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9639,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56424 req_id = 3e0ed7cc-8140-4b4c-b183-e75860b920da
108262025-04-30T06:46:28.275ZDEBGtest_multiple_snapshots (store): query key key = _dendrite._tcp
108272025-04-30T06:46:28.276ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108282025-04-30T06:46:28.277ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56424 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3e0ed7cc-8140-4b4c-b183-e75860b920da
108292025-04-30T06:46:28.277ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108302025-04-30T06:46:28.285ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
108312025-04-30T06:46:28.285ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108322025-04-30T06:46:28.285ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108332025-04-30T06:46:28.286ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108342025-04-30T06:46:28.287ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 42.979364ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
108352025-04-30T06:46:28.287ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108362025-04-30T06:46:28.303ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
108372025-04-30T06:46:28.304ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
108382025-04-30T06:46:28.304ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108392025-04-30T06:46:28.304ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108402025-04-30T06:46:28.305ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
108412025-04-30T06:46:28.306ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108422025-04-30T06:46:28.306ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108432025-04-30T06:46:28.306ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108442025-04-30T06:46:28.312ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
108452025-04-30T06:46:28.329ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
108462025-04-30T06:46:28.329ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108472025-04-30T06:46:28.329ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108482025-04-30T06:46:28.330ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108492025-04-30T06:46:28.335ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
108502025-04-30T06:46:28.335ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
108512025-04-30T06:46:28.335ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108522025-04-30T06:46:28.335ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
108532025-04-30T06:46:28.335ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108542025-04-30T06:46:28.335ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108552025-04-30T06:46:28.337ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4453, tv_nsec: 136068883 } })
108562025-04-30T06:46:28.337ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
108572025-04-30T06:46:28.340ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: d2 ef 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:65145 req_id = 12056f8a-934e-4bf7-997d-6ca42d3b8965
108582025-04-30T06:46:28.340ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53999,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65145 req_id = 12056f8a-934e-4bf7-997d-6ca42d3b8965
108592025-04-30T06:46:28.340ZDEBGtest_multiple_snapshots (store): query key key = _dendrite._tcp
108602025-04-30T06:46:28.340ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108612025-04-30T06:46:28.340ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65145 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 12056f8a-934e-4bf7-997d-6ca42d3b8965
108622025-04-30T06:46:28.341ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4453, tv_nsec: 140790912 } })
108632025-04-30T06:46:28.341ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
108642025-04-30T06:46:28.344ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e c8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63124 req_id = b3fb5331-03c7-4daa-b8da-d83ef25ecdd2
108652025-04-30T06:46:28.344ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40648,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63124 req_id = b3fb5331-03c7-4daa-b8da-d83ef25ecdd2
108662025-04-30T06:46:28.344ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108672025-04-30T06:46:28.344ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63124 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b3fb5331-03c7-4daa-b8da-d83ef25ecdd2
108682025-04-30T06:46:28.344ZINFOtest_multiple_snapshots (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
108692025-04-30T06:46:28.400ZINFOtest_multiple_snapshots (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
108702025-04-30T06:46:28.400ZDEBGtest_multiple_snapshots (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
108712025-04-30T06:46:28.400ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f8 1e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54823 req_id = 5d7cbb1e-f926-4e7c-bbc3-5dae632f972f
108722025-04-30T06:46:28.400ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63518,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54823 req_id = 5d7cbb1e-f926-4e7c-bbc3-5dae632f972f
108732025-04-30T06:46:28.400ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108742025-04-30T06:46:28.400ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54823 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5d7cbb1e-f926-4e7c-bbc3-5dae632f972f
108752025-04-30T06:46:28.402ZINFOtest_multiple_snapshots (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
108762025-04-30T06:46:28.450ZINFOtest_multiple_snapshots (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
108772025-04-30T06:46:28.451ZDEBGtest_multiple_snapshots (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
108782025-04-30T06:46:28.451ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108792025-04-30T06:46:28.460ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108802025-04-30T06:46:28.460ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108812025-04-30T06:46:28.460ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108822025-04-30T06:46:28.461ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108832025-04-30T06:46:28.465ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
108842025-04-30T06:46:28.465ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108852025-04-30T06:46:28.466ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108862025-04-30T06:46:28.477ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
108872025-04-30T06:46:28.482ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 56 f7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 V............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51968 req_id = 7e8bd2be-ace6-401f-811d-5d250feba37a
108882025-04-30T06:46:28.482ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22263,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51968 req_id = 7e8bd2be-ace6-401f-811d-5d250feba37a
108892025-04-30T06:46:28.482ZDEBGtest_multiple_snapshots (store): query key key = _dendrite._tcp
108902025-04-30T06:46:28.483ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108912025-04-30T06:46:28.483ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51968 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e8bd2be-ace6-401f-811d-5d250feba37a
108922025-04-30T06:46:28.483ZINFOtest_multiple_snapshots (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
108932025-04-30T06:46:28.484ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 235.115112ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
108942025-04-30T06:46:28.486ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36409, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4453, tv_nsec: 285011893 } })
108952025-04-30T06:46:28.486ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
108962025-04-30T06:46:28.545ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d0 b3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63217 req_id = 772e5d74-5241-4e9f-98da-71f4c8985f47
108972025-04-30T06:46:28.545ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53427,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63217 req_id = 772e5d74-5241-4e9f-98da-71f4c8985f47
108982025-04-30T06:46:28.545ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108992025-04-30T06:46:28.545ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63217 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 772e5d74-5241-4e9f-98da-71f4c8985f47
109002025-04-30T06:46:28.548ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109012025-04-30T06:46:28.554ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109022025-04-30T06:46:28.554ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109032025-04-30T06:46:28.554ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109042025-04-30T06:46:28.555ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109052025-04-30T06:46:28.558ZINFOtest_multiple_snapshots (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
109062025-04-30T06:46:28.620ZINFOtest_multiple_snapshots (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
109072025-04-30T06:46:28.621ZDEBGtest_multiple_snapshots (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
109082025-04-30T06:46:28.643ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109092025-04-30T06:46:28.643ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109102025-04-30T06:46:28.649ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109112025-04-30T06:46:28.649ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109122025-04-30T06:46:28.649ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109132025-04-30T06:46:28.649ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109142025-04-30T06:46:28.650ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109152025-04-30T06:46:28.652ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109162025-04-30T06:46:28.654ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109172025-04-30T06:46:28.655ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109182025-04-30T06:46:28.655ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
109192025-04-30T06:46:28.658ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109202025-04-30T06:46:28.665ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109212025-04-30T06:46:28.665ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
109222025-04-30T06:46:28.665ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109232025-04-30T06:46:28.665ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109242025-04-30T06:46:28.665ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109252025-04-30T06:46:28.665ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109262025-04-30T06:46:28.677ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109272025-04-30T06:46:28.682ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
109282025-04-30T06:46:28.682ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109292025-04-30T06:46:28.682ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109302025-04-30T06:46:28.682ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109312025-04-30T06:46:28.685ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109322025-04-30T06:46:28.685ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109332025-04-30T06:46:28.685ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 897.936851ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109342025-04-30T06:46:28.685ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
109352025-04-30T06:46:28.685ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109362025-04-30T06:46:28.685ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109372025-04-30T06:46:28.685ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109382025-04-30T06:46:28.685ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109392025-04-30T06:46:28.687ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109402025-04-30T06:46:28.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:54590/v2p
109412025-04-30T06:46:28.803ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109422025-04-30T06:46:28.810ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109432025-04-30T06:46:28.811ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109442025-04-30T06:46:28.811ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109452025-04-30T06:46:28.811ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109462025-04-30T06:46:28.910ZINFOtest_multiple_snapshots (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
109472025-04-30T06:46:28.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:54590/vpc-routes
109482025-04-30T06:46:28.912ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:44161 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109492025-04-30T06:46:28.912ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:36353 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109502025-04-30T06:46:28.919ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
109512025-04-30T06:46:28.919ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
109522025-04-30T06:46:28.923ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:44161 req_id = 243bbc26-4d87-405f-bd2d-1d524db424dc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
109532025-04-30T06:46:28.924ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:36353 req_id = 3f8cd14d-5804-46ab-a9f8-a6b8777b88d8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109542025-04-30T06:46:28.926ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2157 local_addr = [::1]:54590 method = GET remote_addr = [::1]:44161 req_id = 243bbc26-4d87-405f-bd2d-1d524db424dc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
109552025-04-30T06:46:28.926ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2073 local_addr = [::1]:54590 method = GET remote_addr = [::1]:36353 req_id = 3f8cd14d-5804-46ab-a9f8-a6b8777b88d8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
109562025-04-30T06:46:28.929ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
109572025-04-30T06:46:28.931ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 01 8b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54069 req_id = 7d122e34-5893-4e7a-9f68-a7d273963ac7
109582025-04-30T06:46:28.931ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 395,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54069 req_id = 7d122e34-5893-4e7a-9f68-a7d273963ac7
109592025-04-30T06:46:28.931ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
109602025-04-30T06:46:28.931ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109612025-04-30T06:46:28.931ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54069 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7d122e34-5893-4e7a-9f68-a7d273963ac7
109622025-04-30T06:46:28.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:54590/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "243bbc26-4d87-405f-bd2d-1d524db424dc", "content-length": "2", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
109632025-04-30T06:46:28.932ZINFOtest_multiple_snapshots (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
109642025-04-30T06:46:28.932ZINFOtest_multiple_snapshots (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
109652025-04-30T06:46:28.932ZINFOtest_multiple_snapshots (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
109662025-04-30T06:46:28.934ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 687.943482ms iteration = 2 status = {}
109672025-04-30T06:46:28.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:54590/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f8cd14d-5804-46ab-a9f8-a6b8777b88d8", "content-length": "2", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
109682025-04-30T06:46:28.934ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
109692025-04-30T06:46:28.934ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109702025-04-30T06:46:28.935ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109712025-04-30T06:46:28.935ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109722025-04-30T06:46:28.935ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109732025-04-30T06:46:28.936ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109742025-04-30T06:46:28.939ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4453, tv_nsec: 738363476 } })
109752025-04-30T06:46:28.939ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
109762025-04-30T06:46:28.944ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f4 c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60098 req_id = 7813ca1c-4afb-48e5-9c92-bfb134140634
109772025-04-30T06:46:28.944ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62659,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60098 req_id = 7813ca1c-4afb-48e5-9c92-bfb134140634
109782025-04-30T06:46:28.944ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109792025-04-30T06:46:28.945ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60098 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7813ca1c-4afb-48e5-9c92-bfb134140634
109802025-04-30T06:46:29.006ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
109812025-04-30T06:46:29.007ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
109822025-04-30T06:46:29.007ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109832025-04-30T06:46:29.007ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109842025-04-30T06:46:29.007ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109852025-04-30T06:46:29.009ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109862025-04-30T06:46:29.010ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109872025-04-30T06:46:29.011ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109882025-04-30T06:46:29.011ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109892025-04-30T06:46:29.011ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
109902025-04-30T06:46:29.011ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109912025-04-30T06:46:29.012ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109922025-04-30T06:46:29.012ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109932025-04-30T06:46:29.013ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
109942025-04-30T06:46:29.016ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
109952025-04-30T06:46:29.016ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109962025-04-30T06:46:29.033ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
109972025-04-30T06:46:29.033ZERROtest_multiple_snapshots (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
109982025-04-30T06:46:29.033ZERROtest_multiple_snapshots (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
109992025-04-30T06:46:29.033ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 789.299137ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
110002025-04-30T06:46:29.035ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f6 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57300 req_id = c1cbf18e-2499-457d-8061-71f73d7b0bd5
110012025-04-30T06:46:29.036ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63087,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57300 req_id = c1cbf18e-2499-457d-8061-71f73d7b0bd5
110022025-04-30T06:46:29.036ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
110032025-04-30T06:46:29.036ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63087, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57300 req_id = c1cbf18e-2499-457d-8061-71f73d7b0bd5
110042025-04-30T06:46:29.039ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110052025-04-30T06:46:29.046ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110062025-04-30T06:46:29.046ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110072025-04-30T06:46:29.046ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
110082025-04-30T06:46:29.046ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110092025-04-30T06:46:29.047ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
110102025-04-30T06:46:29.047ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
110112025-04-30T06:46:29.049ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
110122025-04-30T06:46:29.049ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
110132025-04-30T06:46:29.049ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
110142025-04-30T06:46:29.052ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110152025-04-30T06:46:29.052ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
110162025-04-30T06:46:29.053ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110172025-04-30T06:46:29.054ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
110182025-04-30T06:46:29.054ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
110192025-04-30T06:46:29.059ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb8abbdf-c66e-45d9-805f-a76158affb77", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110202025-04-30T06:46:29.059ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
110212025-04-30T06:46:29.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42e1ccc-5685-42df-954c-3a85f2af14e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110222025-04-30T06:46:29.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
110232025-04-30T06:46:29.064ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d36159da-958c-4bb0-8f39-55af3eed3f7b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110242025-04-30T06:46:29.064ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
110252025-04-30T06:46:29.068ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bde57086-1c53-45f6-ad3a-3145f9dc9931", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110262025-04-30T06:46:29.068ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
110272025-04-30T06:46:29.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1d3e4f-240b-4e4c-94da-819e686f3229", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110282025-04-30T06:46:29.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
110292025-04-30T06:46:29.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0f5ab63-d54b-4edd-9feb-7c492b219182", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110302025-04-30T06:46:29.086ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
110312025-04-30T06:46:29.122ZINFOtest_multiple_snapshots (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
110322025-04-30T06:46:29.122ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:54590/eip-gateways
110332025-04-30T06:46:29.171ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
110342025-04-30T06:46:29.172ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = PUT remote_addr = [::1]:36353 req_id = 043ef191-7e18-4768-8b90-8d27c0d02638 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110352025-04-30T06:46:29.173ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
110362025-04-30T06:46:29.174ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1164 local_addr = [::1]:54590 method = PUT remote_addr = [::1]:36353 req_id = 043ef191-7e18-4768-8b90-8d27c0d02638 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
110372025-04-30T06:46:29.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:54590/eip-gateways", status: 204, headers: {"x-request-id": "043ef191-7e18-4768-8b90-8d27c0d02638", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110382025-04-30T06:46:29.176ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 925.914093ms iteration = 2 status = {}
110392025-04-30T06:46:29.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "988da187-7b59-4211-b4d6-6eafa7f87bdc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110402025-04-30T06:46:29.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
110412025-04-30T06:46:29.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "277ee9b5-e852-492f-89e2-5397d7e13d38", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110422025-04-30T06:46:29.184ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
110432025-04-30T06:46:29.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3976b90c-24a8-41ba-9cd8-755124a30c2f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110442025-04-30T06:46:29.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
110452025-04-30T06:46:29.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9213410-7f07-487a-9f2c-735d966c97e3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110462025-04-30T06:46:29.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
110472025-04-30T06:46:29.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80b20e13-e0c9-44c4-b9c1-d71ae0f66d13", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110482025-04-30T06:46:29.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
110492025-04-30T06:46:29.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b90f99c9-3947-427a-a4cd-238ae5ad54f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110502025-04-30T06:46:29.190ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
110512025-04-30T06:46:29.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ff106be-2634-476e-96a2-036db3e05fcc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110522025-04-30T06:46:29.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
110532025-04-30T06:46:29.196ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef3a4d78-beaf-481c-9fac-ea43ba6b7f93", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110542025-04-30T06:46:29.196ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
110552025-04-30T06:46:29.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1339d9f4-e84d-44cc-a89c-073892f9f0b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110562025-04-30T06:46:29.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
110572025-04-30T06:46:29.200ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a9c712-5fa4-44fb-a2b1-59ae91ebc621", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110582025-04-30T06:46:29.200ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
110592025-04-30T06:46:29.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ebfaaa2-6576-4c7d-89fb-7b22ec3b698c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110602025-04-30T06:46:29.203ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
110612025-04-30T06:46:29.205ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbfaed85-d455-4fd7-9526-98b6a5a7ccc0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110622025-04-30T06:46:29.205ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
110632025-04-30T06:46:29.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f6b918-6a9a-41c1-a8cb-bae0ee043082", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110642025-04-30T06:46:29.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
110652025-04-30T06:46:29.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14527f16-5cbb-48ff-95ab-6ef0da401e21", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110662025-04-30T06:46:29.209ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
110672025-04-30T06:46:29.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54e8a6b-ad4d-48aa-a008-008f7d4ebfb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110682025-04-30T06:46:29.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
110692025-04-30T06:46:29.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "379c41b5-e447-43f8-9f1a-96743e648d39", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110702025-04-30T06:46:29.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
110712025-04-30T06:46:29.216ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "517103a8-0313-4c90-acc1-e0e9265a33d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110722025-04-30T06:46:29.217ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
110732025-04-30T06:46:29.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e3ce077-0f43-48a8-ad0f-9096b1e081a4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110742025-04-30T06:46:29.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
110752025-04-30T06:46:29.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df44f43-9827-468f-94e6-40ef387cc67c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110762025-04-30T06:46:29.220ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
110772025-04-30T06:46:29.222ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110782025-04-30T06:46:29.222ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110792025-04-30T06:46:29.222ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110802025-04-30T06:46:29.222ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
110812025-04-30T06:46:29.223ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110822025-04-30T06:46:29.224ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3b520b-5275-415f-9ce9-c11f43f47d73", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110832025-04-30T06:46:29.224ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
110842025-04-30T06:46:29.226ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110852025-04-30T06:46:29.233ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110862025-04-30T06:46:29.233ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110872025-04-30T06:46:29.233ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
110882025-04-30T06:46:29.233ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110892025-04-30T06:46:29.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07d1036-2ab5-43e6-a87b-b69469800e82", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110902025-04-30T06:46:29.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
110912025-04-30T06:46:29.240ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110922025-04-30T06:46:29.240ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
110932025-04-30T06:46:29.240ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110942025-04-30T06:46:29.244ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f20ede6-dd5a-48c3-a480-b9fafc0d666b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110952025-04-30T06:46:29.244ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
110962025-04-30T06:46:29.249ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfb39b2b-3d72-49dd-9b31-35002d43e9be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110972025-04-30T06:46:29.250ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
110982025-04-30T06:46:29.258ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "853b2a87-fce9-420b-8cbf-a62ac42d3b25", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
110992025-04-30T06:46:29.258ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
111002025-04-30T06:46:29.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45591094-3d40-4b3a-84e7-9d45fdb724d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111012025-04-30T06:46:29.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
111022025-04-30T06:46:29.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "008924ff-f6da-43e2-bc92-954eb214ea3d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111032025-04-30T06:46:29.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
111042025-04-30T06:46:29.262ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34136978-728b-4f7e-9be6-6b1809aaf837", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111052025-04-30T06:46:29.262ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
111062025-04-30T06:46:29.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d44b59-f8d3-4478-9633-dc91d779e370", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111072025-04-30T06:46:29.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
111082025-04-30T06:46:29.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e7a5702-cd7e-4962-8ba0-29a92c69f5bf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111092025-04-30T06:46:29.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
111102025-04-30T06:46:29.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6952d6a2-8877-4312-b301-b1129d0ebe6a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111112025-04-30T06:46:29.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
111122025-04-30T06:46:29.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "942e935b-1d72-4fe6-9aab-b9d2f36d7cf5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111132025-04-30T06:46:29.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
111142025-04-30T06:46:29.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb1a219-bf06-41a9-9536-73a94899fd0e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111152025-04-30T06:46:29.273ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
111162025-04-30T06:46:29.280ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
111172025-04-30T06:46:29.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "101d2d4b-4ddc-40dd-97cf-ef795fa26e5a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111182025-04-30T06:46:29.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
111192025-04-30T06:46:29.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "008fb3a3-0ff7-4112-bd2b-43705c45027c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111202025-04-30T06:46:29.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
111212025-04-30T06:46:29.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d6da22a-c393-45d7-877e-19b0ae7a4fa3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111222025-04-30T06:46:29.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
111232025-04-30T06:46:29.293ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47ed7b62-2c09-4528-a6cc-49e191ada509", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111242025-04-30T06:46:29.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
111252025-04-30T06:46:29.295ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b7c1c02-7c0c-4982-9981-180e1810b853", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111262025-04-30T06:46:29.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
111272025-04-30T06:46:29.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "126e20d0-9086-4897-8c0e-5da1d6a1fb99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111282025-04-30T06:46:29.297ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
111292025-04-30T06:46:29.299ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dac4607-849a-424b-9098-62b99eb9febd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111302025-04-30T06:46:29.299ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111312025-04-30T06:46:29.299ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111322025-04-30T06:46:29.299ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111332025-04-30T06:46:29.299ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111342025-04-30T06:46:29.300ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111352025-04-30T06:46:29.303ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111362025-04-30T06:46:29.309ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
111372025-04-30T06:46:29.309ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
111382025-04-30T06:46:29.309ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111392025-04-30T06:46:29.309ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111402025-04-30T06:46:29.309ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111412025-04-30T06:46:29.309ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111422025-04-30T06:46:29.312ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111432025-04-30T06:46:29.316ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
111442025-04-30T06:46:29.316ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111452025-04-30T06:46:29.316ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111462025-04-30T06:46:29.317ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111472025-04-30T06:46:29.375ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
111482025-04-30T06:46:29.375ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
111492025-04-30T06:46:29.375ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111502025-04-30T06:46:29.376ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111512025-04-30T06:46:29.376ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111522025-04-30T06:46:29.376ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
111532025-04-30T06:46:29.376ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111542025-04-30T06:46:29.376ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:55737 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111552025-04-30T06:46:29.377ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:55737 req_id = 92d08f5f-f255-4e25-97fa-31f19f01e802 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111562025-04-30T06:46:29.378ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1186 local_addr = [::1]:54590 method = GET remote_addr = [::1]:55737 req_id = 92d08f5f-f255-4e25-97fa-31f19f01e802 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111572025-04-30T06:46:29.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "92d08f5f-f255-4e25-97fa-31f19f01e802", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
111582025-04-30T06:46:29.381ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111592025-04-30T06:46:29.381ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111602025-04-30T06:46:29.382ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
111612025-04-30T06:46:29.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111622025-04-30T06:46:29.383ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111632025-04-30T06:46:29.383ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111642025-04-30T06:46:29.383ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111652025-04-30T06:46:29.384ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111662025-04-30T06:46:29.384ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111672025-04-30T06:46:29.398ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111682025-04-30T06:46:29.398ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111692025-04-30T06:46:29.398ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
111702025-04-30T06:46:29.398ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111712025-04-30T06:46:29.400ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111722025-04-30T06:46:29.405ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111732025-04-30T06:46:29.405ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111742025-04-30T06:46:29.405ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111752025-04-30T06:46:29.406ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111762025-04-30T06:46:29.407ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111772025-04-30T06:46:29.407ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
111782025-04-30T06:46:29.407ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111792025-04-30T06:46:29.509ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
111802025-04-30T06:46:29.509ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
111812025-04-30T06:46:29.510ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 824.4186ms iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
111822025-04-30T06:46:29.510ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
111832025-04-30T06:46:29.510ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111842025-04-30T06:46:29.510ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111852025-04-30T06:46:29.510ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111862025-04-30T06:46:29.510ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111872025-04-30T06:46:29.511ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111882025-04-30T06:46:29.532ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
111892025-04-30T06:46:29.533ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111902025-04-30T06:46:29.541ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111912025-04-30T06:46:29.542ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111922025-04-30T06:46:29.542ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111932025-04-30T06:46:29.543ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111942025-04-30T06:46:29.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
111952025-04-30T06:46:29.549ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111962025-04-30T06:46:29.549ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 07574691-668c-4b6a-810b-702427db72d1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111972025-04-30T06:46:29.549ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 07574691-668c-4b6a-810b-702427db72d1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111982025-04-30T06:46:29.549ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 07574691-668c-4b6a-810b-702427db72d1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111992025-04-30T06:46:29.550ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 07574691-668c-4b6a-810b-702427db72d1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112002025-04-30T06:46:29.550ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
112012025-04-30T06:46:29.550ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112022025-04-30T06:46:29.552ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cc 12 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61871 req_id = ba67a369-6331-476d-8955-ee4effa5d95f
112032025-04-30T06:46:29.552ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52242,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61871 req_id = ba67a369-6331-476d-8955-ee4effa5d95f
112042025-04-30T06:46:29.552ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
112052025-04-30T06:46:29.553ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112062025-04-30T06:46:29.553ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61871 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ba67a369-6331-476d-8955-ee4effa5d95f
112072025-04-30T06:46:29.553ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 4642 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 07574691-668c-4b6a-810b-702427db72d1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112082025-04-30T06:46:29.554ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4454, tv_nsec: 353412581 } })
112092025-04-30T06:46:29.554ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
112102025-04-30T06:46:29.555ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "07574691-668c-4b6a-810b-702427db72d1", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
112112025-04-30T06:46:29.555ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112122025-04-30T06:46:29.557ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112132025-04-30T06:46:29.557ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112142025-04-30T06:46:29.557ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112152025-04-30T06:46:29.560ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112162025-04-30T06:46:29.561ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d 8c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49699 req_id = 99de18df-62f6-4cc0-8974-1b9675acc19e
112172025-04-30T06:46:29.561ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32140,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49699 req_id = 99de18df-62f6-4cc0-8974-1b9675acc19e
112182025-04-30T06:46:29.561ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112192025-04-30T06:46:29.561ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49699 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 99de18df-62f6-4cc0-8974-1b9675acc19e
112202025-04-30T06:46:29.610ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112212025-04-30T06:46:29.611ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 54158 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112222025-04-30T06:46:29.611ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112232025-04-30T06:46:29.611ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112242025-04-30T06:46:29.611ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112252025-04-30T06:46:29.611ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
112262025-04-30T06:46:29.612ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112272025-04-30T06:46:29.612ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112282025-04-30T06:46:29.612ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ddc0e3e-71e2-4b4e-acc0-d372c2bb26dd", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
112292025-04-30T06:46:29.613ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112302025-04-30T06:46:29.613ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112312025-04-30T06:46:29.613ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
112322025-04-30T06:46:29.613ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112332025-04-30T06:46:29.613ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = 127.0.0.1:51104 method = GET remote_addr = 127.0.0.1:62425 req_id = 0e6b3bfd-3df8-464f-917f-0548dc6d45a9 uri = /3ae6adf1-675a-4c6a-a06e-473e2878df6e
112342025-04-30T06:46:29.613ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6a 47 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c jG..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61968 req_id = b9287423-7991-43ff-a983-f48567033e67
112352025-04-30T06:46:29.614ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27207,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61968 req_id = b9287423-7991-43ff-a983-f48567033e67
112362025-04-30T06:46:29.614ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
112372025-04-30T06:46:29.614ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27207, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61968 req_id = b9287423-7991-43ff-a983-f48567033e67
112382025-04-30T06:46:29.618ZINFOtest_multiple_snapshots (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4305 local_addr = 127.0.0.1:51104 method = GET remote_addr = 127.0.0.1:62425 req_id = 0e6b3bfd-3df8-464f-917f-0548dc6d45a9 response_code = 200 uri = /3ae6adf1-675a-4c6a-a06e-473e2878df6e
112392025-04-30T06:46:29.618ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
112402025-04-30T06:46:29.618ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
112412025-04-30T06:46:29.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
112422025-04-30T06:46:29.619ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112432025-04-30T06:46:29.624ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112442025-04-30T06:46:29.624ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112452025-04-30T06:46:29.624ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
112462025-04-30T06:46:29.624ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112472025-04-30T06:46:29.631ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112482025-04-30T06:46:29.631ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112492025-04-30T06:46:29.631ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
112502025-04-30T06:46:29.631ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112512025-04-30T06:46:29.635ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112522025-04-30T06:46:29.635ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
112532025-04-30T06:46:29.636ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112542025-04-30T06:46:29.636ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112552025-04-30T06:46:29.636ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "64b639da-5ec1-42d9-be83-75004beae3d7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
112562025-04-30T06:46:29.636ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
112572025-04-30T06:46:29.636ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112582025-04-30T06:46:29.638ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112592025-04-30T06:46:29.639ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112602025-04-30T06:46:29.639ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112612025-04-30T06:46:29.639ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112622025-04-30T06:46:29.640ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6679cc8a-10e7-4828-8beb-ef4cc3decbe0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
112632025-04-30T06:46:29.641ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
112642025-04-30T06:46:29.644ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f314bdb4-2c17-4295-9953-e78f27e0ae9f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:28 GMT"} })
112652025-04-30T06:46:29.644ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
112662025-04-30T06:46:29.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fc11979-7b85-44be-8d57-c617322d3050", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112672025-04-30T06:46:29.648ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
112682025-04-30T06:46:29.651ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca73dc1d-b687-4ef7-84f7-8b7f0b74da22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112692025-04-30T06:46:29.651ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
112702025-04-30T06:46:29.652ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93036384-bce9-459d-adfa-56f8d155704f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112712025-04-30T06:46:29.652ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
112722025-04-30T06:46:29.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b468b154-4609-42c8-a444-bfb5e90b74f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112732025-04-30T06:46:29.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
112742025-04-30T06:46:29.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47d4ea8c-721b-4118-8f95-a27054dfa3ab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112752025-04-30T06:46:29.657ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
112762025-04-30T06:46:29.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ae5f63b-e03f-43e9-b8f0-5b7b007273fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112772025-04-30T06:46:29.658ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
112782025-04-30T06:46:29.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2d5978-f6d4-4e50-8eb4-53405ee9f1e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112792025-04-30T06:46:29.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
112802025-04-30T06:46:29.662ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef009cc5-162b-4bde-b0e9-4e00968a6e3e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112812025-04-30T06:46:29.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
112822025-04-30T06:46:29.664ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e7ab1ea-5d56-4179-945f-5b550c0db57d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112832025-04-30T06:46:29.664ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
112842025-04-30T06:46:29.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42426d3b-7f4e-4048-8460-4427664499fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112852025-04-30T06:46:29.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
112862025-04-30T06:46:29.668ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6901301c-9366-4fd5-b7ea-1955f7757430", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112872025-04-30T06:46:29.668ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
112882025-04-30T06:46:29.672ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4728b5f6-acae-484c-8696-4816dde14659", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112892025-04-30T06:46:29.673ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
112902025-04-30T06:46:29.677ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
112912025-04-30T06:46:29.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f08a11-b144-4e3e-97c8-c12342c63449", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112922025-04-30T06:46:29.678ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
112932025-04-30T06:46:29.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc5d1a6-89f8-4733-9780-16d44040f759", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112942025-04-30T06:46:29.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
112952025-04-30T06:46:29.683ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2d41429-0e9c-4d02-85d1-dada44ac171d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112962025-04-30T06:46:29.683ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
112972025-04-30T06:46:29.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b27af40a-4cdb-4bab-8cab-5f2afc30a5c9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
112982025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
112992025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8294d961-63b1-4e41-92d7-f2b015957f07", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113002025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
113012025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46013414-2567-44a2-a1d8-e828c25ad03f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113022025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
113032025-04-30T06:46:29.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c883db09-6af1-4267-8ebe-485814267b82", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113042025-04-30T06:46:29.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
113052025-04-30T06:46:29.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c47e2a84-9b98-4596-b581-bc15019bbd8a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113062025-04-30T06:46:29.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
113072025-04-30T06:46:29.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5358e927-fc3f-41ec-8d0b-a718f4e50428", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113082025-04-30T06:46:29.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
113092025-04-30T06:46:29.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "133211cc-68cb-4859-9af1-d25580b2b6f0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113102025-04-30T06:46:29.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
113112025-04-30T06:46:29.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14c35a53-895f-40c9-84ae-cd92a2babef4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113122025-04-30T06:46:29.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
113132025-04-30T06:46:29.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd38b100-ad6d-43cf-abc1-61d3ab563558", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113142025-04-30T06:46:29.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
113152025-04-30T06:46:29.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa1fc91a-e73e-4c4d-a6dc-84d01e9a303f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113162025-04-30T06:46:29.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
113172025-04-30T06:46:29.707ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55cd461f-caae-477f-aabc-3182367afadf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113182025-04-30T06:46:29.707ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
113192025-04-30T06:46:29.710ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1aec13e-6d18-47a8-ade2-393fcc4a7d51", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113202025-04-30T06:46:29.710ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
113212025-04-30T06:46:29.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e387ebbc-82f9-482d-876f-34af47382108", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113222025-04-30T06:46:29.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
113232025-04-30T06:46:29.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e51075d-eef7-43ad-82f6-563a92dc5cf9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113242025-04-30T06:46:29.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
113252025-04-30T06:46:29.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "917d7bf4-46cf-4bc0-a3a5-2222121a3c10", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113262025-04-30T06:46:29.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
113272025-04-30T06:46:29.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e3ee0b4-eac2-41c1-a621-1e2c85174dd2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113282025-04-30T06:46:29.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
113292025-04-30T06:46:29.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1f4cef-4280-4036-a052-ff336362c2ac", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113302025-04-30T06:46:29.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
113312025-04-30T06:46:29.723ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cc7277e-479b-4627-9fda-c6a714a64708", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113322025-04-30T06:46:29.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
113332025-04-30T06:46:29.730ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113342025-04-30T06:46:29.730ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113352025-04-30T06:46:29.730ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113362025-04-30T06:46:29.730ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
113372025-04-30T06:46:29.731ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113382025-04-30T06:46:29.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7de0fbd-35c1-4df4-aae0-ee49137a5f31", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113392025-04-30T06:46:29.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
113402025-04-30T06:46:29.737ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113412025-04-30T06:46:29.752ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113422025-04-30T06:46:29.752ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113432025-04-30T06:46:29.752ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
113442025-04-30T06:46:29.753ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113452025-04-30T06:46:29.760ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b9af87-fc86-439d-a870-41fafa657f92", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113462025-04-30T06:46:29.761ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
113472025-04-30T06:46:29.762ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113482025-04-30T06:46:29.763ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
113492025-04-30T06:46:29.763ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113502025-04-30T06:46:29.765ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10503e0d-1b25-43e7-b42d-29bfc48775ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113512025-04-30T06:46:29.765ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
113522025-04-30T06:46:29.768ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae8c95dc-1d82-40ea-9b53-7b4a60fced2c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113532025-04-30T06:46:29.768ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
113542025-04-30T06:46:29.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b1c8a4e-ff0c-47dc-81c6-6a390522ac86", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113552025-04-30T06:46:29.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
113562025-04-30T06:46:29.772ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "739fc6fc-a428-4fb7-a28d-bf29c6577c8b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113572025-04-30T06:46:29.772ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
113582025-04-30T06:46:29.773ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca3e246c-dd10-4a00-b13e-7bff9d5e86d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113592025-04-30T06:46:29.773ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
113602025-04-30T06:46:29.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdf884b9-e082-48dd-9eb6-6fc269675898", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113612025-04-30T06:46:29.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
113622025-04-30T06:46:29.776ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0cdb50e-20a0-4333-bdbe-88f5cd6f23b6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113632025-04-30T06:46:29.776ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113642025-04-30T06:46:29.777ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113652025-04-30T06:46:29.777ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113662025-04-30T06:46:29.777ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113672025-04-30T06:46:29.777ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113682025-04-30T06:46:29.783ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113692025-04-30T06:46:29.790ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113702025-04-30T06:46:29.790ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113712025-04-30T06:46:29.790ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113722025-04-30T06:46:29.791ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113732025-04-30T06:46:29.791ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113742025-04-30T06:46:29.793ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113752025-04-30T06:46:29.801ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113762025-04-30T06:46:29.811ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113772025-04-30T06:46:29.811ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113782025-04-30T06:46:29.811ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113792025-04-30T06:46:29.811ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113802025-04-30T06:46:29.900ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
113812025-04-30T06:46:29.901ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
113822025-04-30T06:46:29.904ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:61039 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113832025-04-30T06:46:29.905ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:61039 req_id = 5bd7042b-2804-4f35-970f-b89fb3c3916d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113842025-04-30T06:46:29.905ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 509 local_addr = [::1]:54590 method = GET remote_addr = [::1]:61039 req_id = 5bd7042b-2804-4f35-970f-b89fb3c3916d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113852025-04-30T06:46:29.906ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
113862025-04-30T06:46:29.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bd7042b-2804-4f35-970f-b89fb3c3916d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
113872025-04-30T06:46:29.909ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113882025-04-30T06:46:29.909ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113892025-04-30T06:46:29.909ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
113902025-04-30T06:46:29.911ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113912025-04-30T06:46:29.911ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113922025-04-30T06:46:29.911ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113932025-04-30T06:46:29.911ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113942025-04-30T06:46:29.912ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113952025-04-30T06:46:29.916ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113962025-04-30T06:46:29.920ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113972025-04-30T06:46:29.921ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113982025-04-30T06:46:29.921ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113992025-04-30T06:46:29.922ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114002025-04-30T06:46:29.960ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114012025-04-30T06:46:29.969ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114022025-04-30T06:46:29.969ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114032025-04-30T06:46:29.969ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
114042025-04-30T06:46:29.969ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114052025-04-30T06:46:29.974ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
114062025-04-30T06:46:29.975ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
114072025-04-30T06:46:29.975ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 464.935844ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
114082025-04-30T06:46:29.975ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
114092025-04-30T06:46:29.975ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114102025-04-30T06:46:29.975ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114112025-04-30T06:46:29.975ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114122025-04-30T06:46:29.975ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114132025-04-30T06:46:29.975ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114142025-04-30T06:46:29.979ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114152025-04-30T06:46:29.986ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114162025-04-30T06:46:29.986ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114172025-04-30T06:46:29.986ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
114182025-04-30T06:46:29.987ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114192025-04-30T06:46:29.987ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114202025-04-30T06:46:29.992ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114212025-04-30T06:46:29.993ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114222025-04-30T06:46:29.993ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114232025-04-30T06:46:29.993ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114242025-04-30T06:46:29.994ZDEBGtest_multiple_snapshots (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114252025-04-30T06:46:29.994ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114262025-04-30T06:46:29.996ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
114272025-04-30T06:46:30.011ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
114282025-04-30T06:46:30.012ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
114292025-04-30T06:46:30.012ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
114302025-04-30T06:46:30.013ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114312025-04-30T06:46:30.013ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
114322025-04-30T06:46:30.014ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114332025-04-30T06:46:30.017ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 12 table_name = fields_string
114342025-04-30T06:46:30.017ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
114352025-04-30T06:46:30.018ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f2 e9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50342 req_id = bd36d29b-08f0-4abe-b41a-69726a91e40a
114362025-04-30T06:46:30.018ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62185,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50342 req_id = bd36d29b-08f0-4abe-b41a-69726a91e40a
114372025-04-30T06:46:30.018ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
114382025-04-30T06:46:30.018ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114392025-04-30T06:46:30.019ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50342 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bd36d29b-08f0-4abe-b41a-69726a91e40a
114402025-04-30T06:46:30.020ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4454, tv_nsec: 818958531 } })
114412025-04-30T06:46:30.020ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
114422025-04-30T06:46:30.022ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 5 table_name = fields_u16
114432025-04-30T06:46:30.022ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
114442025-04-30T06:46:30.026ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d9 e5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62419 req_id = ef88768d-ddc4-4076-9e96-c88154892aa0
114452025-04-30T06:46:30.026ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55781,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62419 req_id = ef88768d-ddc4-4076-9e96-c88154892aa0
114462025-04-30T06:46:30.026ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114472025-04-30T06:46:30.027ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62419 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ef88768d-ddc4-4076-9e96-c88154892aa0
114482025-04-30T06:46:30.079ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
114492025-04-30T06:46:30.080ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 5 table_name = fields_uuid
114502025-04-30T06:46:30.080ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
114512025-04-30T06:46:30.082ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 23 2b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c #+..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53482 req_id = e95ad0db-06b0-467d-b2b7-046779cb5be7
114522025-04-30T06:46:30.082ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9003,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53482 req_id = e95ad0db-06b0-467d-b2b7-046779cb5be7
114532025-04-30T06:46:30.083ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
114542025-04-30T06:46:30.083ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9003, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53482 req_id = e95ad0db-06b0-467d-b2b7-046779cb5be7
114552025-04-30T06:46:30.084ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
114562025-04-30T06:46:30.084ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
114572025-04-30T06:46:30.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
114582025-04-30T06:46:30.090ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 5 table_name = measurements_histogramu64
114592025-04-30T06:46:30.091ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
114602025-04-30T06:46:30.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b15e087-1daa-4915-b2a0-16157658b638", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114612025-04-30T06:46:30.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
114622025-04-30T06:46:30.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e82335-980f-4ac0-b47b-0c6df504b4cd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114632025-04-30T06:46:30.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
114642025-04-30T06:46:30.096ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
114652025-04-30T06:46:30.096ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114662025-04-30T06:46:30.096ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
114672025-04-30T06:46:30.101ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbae1309-baf4-4999-ac41-a4f1583be550", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114682025-04-30T06:46:30.101ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
114692025-04-30T06:46:30.103ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abc2d93-de1d-4534-9b38-0a00f8ace434", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114702025-04-30T06:46:30.103ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
114712025-04-30T06:46:30.106ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
114722025-04-30T06:46:30.108ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b405bec1-1327-45f1-9d3f-1e8f177b63a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114732025-04-30T06:46:30.109ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
114742025-04-30T06:46:30.111ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f968a89-4495-4d2d-9142-22b606b0494f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114752025-04-30T06:46:30.111ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
114762025-04-30T06:46:30.114ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f8f596-48ab-4fcd-8496-e77c1f3d03f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114772025-04-30T06:46:30.114ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
114782025-04-30T06:46:30.120ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63861f72-c448-494f-8d60-530a76cf5d96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114792025-04-30T06:46:30.121ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
114802025-04-30T06:46:30.122ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2272591-4827-4181-a6ad-94200aeedb56", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114812025-04-30T06:46:30.122ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
114822025-04-30T06:46:30.123ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "580307a9-942c-4482-b1e2-1b81e59dc459", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114832025-04-30T06:46:30.124ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
114842025-04-30T06:46:30.125ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1628e187-3139-4990-8812-14abd47f4056", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114852025-04-30T06:46:30.125ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
114862025-04-30T06:46:30.127ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40846bcd-f260-43bc-814e-1e5464249e1d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114872025-04-30T06:46:30.127ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
114882025-04-30T06:46:30.128ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbce2c18-f897-4ad9-bbea-259ed6976813", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114892025-04-30T06:46:30.128ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
114902025-04-30T06:46:30.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40fa9818-1d20-4685-a142-cee85cf32ba1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114912025-04-30T06:46:30.131ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
114922025-04-30T06:46:30.133ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a01c601-875d-4cec-9456-a2d9353ae427", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114932025-04-30T06:46:30.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
114942025-04-30T06:46:30.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9805557b-4ea6-4047-92e4-0230997ff386", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114952025-04-30T06:46:30.136ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
114962025-04-30T06:46:30.138ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd0b1ee8-c9c7-4ac6-a6c6-ba0c7a97d8f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114972025-04-30T06:46:30.138ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
114982025-04-30T06:46:30.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "212b8135-afe8-4215-bd38-01d45205643c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
114992025-04-30T06:46:30.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
115002025-04-30T06:46:30.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b75e3a71-a61a-4a99-acc1-651975898bb1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115012025-04-30T06:46:30.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
115022025-04-30T06:46:30.145ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5438df8e-b219-40a2-8460-89dd90db09c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115032025-04-30T06:46:30.145ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
115042025-04-30T06:46:30.146ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4134dc3-0c32-4059-a361-1ed8ec5c13f4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115052025-04-30T06:46:30.147ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
115062025-04-30T06:46:30.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c862584c-68bb-4d3c-a23d-b482b0be268f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115072025-04-30T06:46:30.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
115082025-04-30T06:46:30.151ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a22f3181-b18e-4e54-a070-eeee5f0a0a7a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115092025-04-30T06:46:30.151ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
115102025-04-30T06:46:30.152ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9cf7f80-e699-4333-b1e2-168973574b9b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115112025-04-30T06:46:30.152ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
115122025-04-30T06:46:30.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f59ec6d-7b97-4b73-887f-cc536c02befa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115132025-04-30T06:46:30.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
115142025-04-30T06:46:30.158ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115152025-04-30T06:46:30.158ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115162025-04-30T06:46:30.158ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115172025-04-30T06:46:30.158ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
115182025-04-30T06:46:30.160ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115192025-04-30T06:46:30.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "634cadee-f992-4c62-9ee0-41b56f61083d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115202025-04-30T06:46:30.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
115212025-04-30T06:46:30.163ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115222025-04-30T06:46:30.170ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115232025-04-30T06:46:30.170ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115242025-04-30T06:46:30.170ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
115252025-04-30T06:46:30.171ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115262025-04-30T06:46:30.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d3064aa-0cf3-4f19-ae18-f43db2a4036c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115272025-04-30T06:46:30.171ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
115282025-04-30T06:46:30.173ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115292025-04-30T06:46:30.173ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
115302025-04-30T06:46:30.174ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115312025-04-30T06:46:30.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f0fbae-959d-485c-8dd5-61409f214ba1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115322025-04-30T06:46:30.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
115332025-04-30T06:46:30.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afbfdf3f-bd58-4003-8cba-f03bd301f2d3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115342025-04-30T06:46:30.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
115352025-04-30T06:46:30.180ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "695e2810-94ad-4f4b-b55d-2c27f233d10d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115362025-04-30T06:46:30.180ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
115372025-04-30T06:46:30.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79a6b025-83df-4306-8561-7d1e278ff4a3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115382025-04-30T06:46:30.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
115392025-04-30T06:46:30.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc494561-67f0-429a-80bc-5cb4664f4b65", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115402025-04-30T06:46:30.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
115412025-04-30T06:46:30.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "466ea53e-121b-4de1-9770-bd794a0331e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115422025-04-30T06:46:30.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
115432025-04-30T06:46:30.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8f4beb3-cdbe-4d03-9e35-0ee33bfe575f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115442025-04-30T06:46:30.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
115452025-04-30T06:46:30.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "648fc2f0-534f-4283-bd91-ed469595337d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115462025-04-30T06:46:30.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
115472025-04-30T06:46:30.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d66f9d36-af38-41b9-a741-4d28d20f4e48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115482025-04-30T06:46:30.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
115492025-04-30T06:46:30.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05836a9b-dc8a-4484-93f3-bf6c93b3b492", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115502025-04-30T06:46:30.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
115512025-04-30T06:46:30.201ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77666707-9dc4-4f71-b6ee-a830f545697e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115522025-04-30T06:46:30.202ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
115532025-04-30T06:46:30.202ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
115542025-04-30T06:46:30.204ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc88d7d4-031f-469a-a6d9-f2b89cb2439d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115552025-04-30T06:46:30.204ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
115562025-04-30T06:46:30.207ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda87a22-2960-4822-80c9-03cabfd396e6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115572025-04-30T06:46:30.207ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
115582025-04-30T06:46:30.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe82a58-0419-40c7-b5e3-c6123f90af59", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115592025-04-30T06:46:30.210ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
115602025-04-30T06:46:30.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "707a26d9-6f46-44ea-8337-39dbc175dc15", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115612025-04-30T06:46:30.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
115622025-04-30T06:46:30.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8fe07f-47f1-4d78-8b7e-0368e14efb78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115632025-04-30T06:46:30.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
115642025-04-30T06:46:30.213ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddf76540-f1e5-43c9-ad71-856cb0ee6448", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115652025-04-30T06:46:30.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
115662025-04-30T06:46:30.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "163d3642-68cb-4606-914e-425c8572f392", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115672025-04-30T06:46:30.215ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115682025-04-30T06:46:30.215ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115692025-04-30T06:46:30.215ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115702025-04-30T06:46:30.215ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115712025-04-30T06:46:30.216ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115722025-04-30T06:46:30.219ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115732025-04-30T06:46:30.228ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115742025-04-30T06:46:30.228ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115752025-04-30T06:46:30.228ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115762025-04-30T06:46:30.228ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115772025-04-30T06:46:30.228ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115782025-04-30T06:46:30.229ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115792025-04-30T06:46:30.231ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115802025-04-30T06:46:30.237ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115812025-04-30T06:46:30.237ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115822025-04-30T06:46:30.237ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115832025-04-30T06:46:30.238ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115842025-04-30T06:46:30.298ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
115852025-04-30T06:46:30.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
115862025-04-30T06:46:30.299ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115872025-04-30T06:46:30.299ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115882025-04-30T06:46:30.299ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115892025-04-30T06:46:30.299ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
115902025-04-30T06:46:30.300ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115912025-04-30T06:46:30.300ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:37657 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
115922025-04-30T06:46:30.301ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:37657 req_id = 394a4a05-fbd5-4258-9500-843096ada5ca sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115932025-04-30T06:46:30.301ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 461 local_addr = [::1]:54590 method = GET remote_addr = [::1]:37657 req_id = 394a4a05-fbd5-4258-9500-843096ada5ca response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115942025-04-30T06:46:30.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "394a4a05-fbd5-4258-9500-843096ada5ca", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
115952025-04-30T06:46:30.304ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115962025-04-30T06:46:30.304ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115972025-04-30T06:46:30.304ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
115982025-04-30T06:46:30.306ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115992025-04-30T06:46:30.306ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116002025-04-30T06:46:30.306ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116012025-04-30T06:46:30.306ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116022025-04-30T06:46:30.307ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116032025-04-30T06:46:30.308ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116042025-04-30T06:46:30.314ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116052025-04-30T06:46:30.314ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116062025-04-30T06:46:30.314ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
116072025-04-30T06:46:30.317ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116082025-04-30T06:46:30.319ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116092025-04-30T06:46:30.342ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116102025-04-30T06:46:30.342ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116112025-04-30T06:46:30.342ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116122025-04-30T06:46:30.343ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116132025-04-30T06:46:30.343ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116142025-04-30T06:46:30.343ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
116152025-04-30T06:46:30.344ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116162025-04-30T06:46:30.397ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
116172025-04-30T06:46:30.398ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
116182025-04-30T06:46:30.398ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 422.993036ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
116192025-04-30T06:46:30.398ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
116202025-04-30T06:46:30.398ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
116212025-04-30T06:46:30.398ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116222025-04-30T06:46:30.398ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116232025-04-30T06:46:30.398ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116242025-04-30T06:46:30.400ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116252025-04-30T06:46:30.402ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
116262025-04-30T06:46:30.410ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116272025-04-30T06:46:30.416ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116282025-04-30T06:46:30.416ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116292025-04-30T06:46:30.416ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116302025-04-30T06:46:30.416ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116312025-04-30T06:46:30.424ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
116322025-04-30T06:46:30.424ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
116332025-04-30T06:46:30.429ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bd 84 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60043 req_id = 61405924-c205-4cea-8ad0-fc24288221a5
116342025-04-30T06:46:30.429ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48516,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60043 req_id = 61405924-c205-4cea-8ad0-fc24288221a5
116352025-04-30T06:46:30.430ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
116362025-04-30T06:46:30.430ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116372025-04-30T06:46:30.430ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60043 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 61405924-c205-4cea-8ad0-fc24288221a5
116382025-04-30T06:46:30.431ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4455, tv_nsec: 230337207 } })
116392025-04-30T06:46:30.431ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
116402025-04-30T06:46:30.437ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53851 req_id = a471fe87-03e1-489a-9fa9-1de3714b27c9
116412025-04-30T06:46:30.437ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7170,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53851 req_id = a471fe87-03e1-489a-9fa9-1de3714b27c9
116422025-04-30T06:46:30.437ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116432025-04-30T06:46:30.438ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53851 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a471fe87-03e1-489a-9fa9-1de3714b27c9
116442025-04-30T06:46:30.500ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
116452025-04-30T06:46:30.503ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116462025-04-30T06:46:30.503ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116472025-04-30T06:46:30.504ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116482025-04-30T06:46:30.504ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
116492025-04-30T06:46:30.504ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116502025-04-30T06:46:30.509ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e8 72 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .r..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61520 req_id = cb34d797-202d-494b-a73b-4ae2072525a1
116512025-04-30T06:46:30.509ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59506,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61520 req_id = cb34d797-202d-494b-a73b-4ae2072525a1
116522025-04-30T06:46:30.510ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
116532025-04-30T06:46:30.510ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59506, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61520 req_id = cb34d797-202d-494b-a73b-4ae2072525a1
116542025-04-30T06:46:30.511ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
116552025-04-30T06:46:30.511ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
116562025-04-30T06:46:30.511ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
116572025-04-30T06:46:30.513ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116582025-04-30T06:46:30.521ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116592025-04-30T06:46:30.521ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116602025-04-30T06:46:30.521ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
116612025-04-30T06:46:30.521ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116622025-04-30T06:46:30.529ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116632025-04-30T06:46:30.529ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
116642025-04-30T06:46:30.531ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116652025-04-30T06:46:30.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af0d05d1-efdf-4b46-be25-15700b7dfbe9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116662025-04-30T06:46:30.542ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
116672025-04-30T06:46:30.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7d83b3-cfdd-495a-a1ec-8306568e377e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116682025-04-30T06:46:30.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
116692025-04-30T06:46:30.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7b81f65-1066-43a3-b894-c13360e9bc78", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116702025-04-30T06:46:30.554ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
116712025-04-30T06:46:30.562ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "858f8466-0a5a-4b6f-87f1-485a51cc04f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116722025-04-30T06:46:30.562ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
116732025-04-30T06:46:30.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e1d6cd5-6a96-40c7-9723-fe3c32f1e055", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116742025-04-30T06:46:30.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
116752025-04-30T06:46:30.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc4feea2-361d-4806-8a43-e8bc43f97297", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116762025-04-30T06:46:30.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
116772025-04-30T06:46:30.581ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
116782025-04-30T06:46:30.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b96c89f-1652-4719-beab-3d3c411e347d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116792025-04-30T06:46:30.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
116802025-04-30T06:46:30.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd155ecb-bae2-4c2b-904f-3e2b923a8e29", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116812025-04-30T06:46:30.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
116822025-04-30T06:46:30.590ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be308df3-6ec4-41c7-8776-eb4777704d8d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116832025-04-30T06:46:30.590ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
116842025-04-30T06:46:30.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1065657-c25f-4c20-8069-f5ded36fd274", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116852025-04-30T06:46:30.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
116862025-04-30T06:46:30.595ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d93610cb-8927-4162-aefc-c5394b3fe0ff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116872025-04-30T06:46:30.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
116882025-04-30T06:46:30.597ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf212b5-c29e-482c-a664-0f764b24f993", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116892025-04-30T06:46:30.597ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
116902025-04-30T06:46:30.599ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3d00f9-d5d7-48d4-a0fc-204cdc036056", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116912025-04-30T06:46:30.599ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
116922025-04-30T06:46:30.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad017e8e-e092-4bc1-924c-b82209843617", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116932025-04-30T06:46:30.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
116942025-04-30T06:46:30.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e7e170e-5f3a-4e0a-89ba-503566cc3f81", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116952025-04-30T06:46:30.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
116962025-04-30T06:46:30.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "227fc181-cb16-4b33-ba77-d4e8ed0d274c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116972025-04-30T06:46:30.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
116982025-04-30T06:46:30.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c60dc0f6-c5ff-4ad0-a95f-8274d37da315", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
116992025-04-30T06:46:30.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
117002025-04-30T06:46:30.613ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a05a9ccf-d189-4676-982e-d84e732fef8a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117012025-04-30T06:46:30.613ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
117022025-04-30T06:46:30.619ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d28c7cb5-4340-4816-aba2-e9ed0c88e505", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117032025-04-30T06:46:30.620ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
117042025-04-30T06:46:30.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb67c28-5acf-4bfa-a3af-20fb4f4edd08", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117052025-04-30T06:46:30.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
117062025-04-30T06:46:30.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e9a9b42-88e1-4e76-b8e7-edde4e4e6160", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117072025-04-30T06:46:30.623ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
117082025-04-30T06:46:30.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e12421c9-c645-4c4f-9819-c96984bffbb5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117092025-04-30T06:46:30.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
117102025-04-30T06:46:30.634ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117112025-04-30T06:46:30.634ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117122025-04-30T06:46:30.634ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117132025-04-30T06:46:30.634ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
117142025-04-30T06:46:30.634ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117152025-04-30T06:46:30.634ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "211a7779-e82e-4f2f-b2f9-53468bb960aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117162025-04-30T06:46:30.634ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
117172025-04-30T06:46:30.636ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117182025-04-30T06:46:30.636ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3912f1-a524-4bee-af80-187c02a6ec1b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117192025-04-30T06:46:30.637ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
117202025-04-30T06:46:30.637ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117212025-04-30T06:46:30.638ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117222025-04-30T06:46:30.639ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117232025-04-30T06:46:30.639ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117242025-04-30T06:46:30.639ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117252025-04-30T06:46:30.640ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d4eca73-4354-40be-9be6-268d10daa917", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:29 GMT"} })
117262025-04-30T06:46:30.640ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
117272025-04-30T06:46:30.641ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117282025-04-30T06:46:30.656ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117292025-04-30T06:46:30.657ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117302025-04-30T06:46:30.657ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
117312025-04-30T06:46:30.657ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117322025-04-30T06:46:30.661ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117332025-04-30T06:46:30.661ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
117342025-04-30T06:46:30.661ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117352025-04-30T06:46:30.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7881d51-7ba8-4655-8f09-f3a9da5c5bc7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117362025-04-30T06:46:30.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
117372025-04-30T06:46:30.666ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "108eeb82-a2c3-4a68-9e91-5729b225b050", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117382025-04-30T06:46:30.667ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
117392025-04-30T06:46:30.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbfa4676-28cb-4a04-8e2b-8dcb4d1e4364", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117402025-04-30T06:46:30.674ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
117412025-04-30T06:46:30.679ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d12a78-216d-4ea9-9a79-28966a9910ce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117422025-04-30T06:46:30.679ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
117432025-04-30T06:46:30.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efd8b7cd-5a2b-4854-89a0-456535cf5f92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117442025-04-30T06:46:30.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
117452025-04-30T06:46:30.686ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "612f79f2-618b-4a1d-bf07-690185de9626", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117462025-04-30T06:46:30.686ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
117472025-04-30T06:46:30.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea25be1c-3454-4dca-8c15-3fe990f4d583", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117482025-04-30T06:46:30.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
117492025-04-30T06:46:30.689ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ca7dc0d-357c-4cf1-856b-24e6814f3cb0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117502025-04-30T06:46:30.689ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
117512025-04-30T06:46:30.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8925d2e-3339-43f9-873f-83b3d74b8d2e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117522025-04-30T06:46:30.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
117532025-04-30T06:46:30.697ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
117542025-04-30T06:46:30.700ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2138e2d5-d450-41dc-b6f7-6ee17498f7ed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117552025-04-30T06:46:30.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
117562025-04-30T06:46:30.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9accc14a-992b-4e90-acf8-a661ac54a74b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117572025-04-30T06:46:30.706ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
117582025-04-30T06:46:30.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec7882e7-7392-4674-a4a0-3ec81541e8e4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117592025-04-30T06:46:30.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
117602025-04-30T06:46:30.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2171bf-d237-486c-87fe-829330bc718d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117612025-04-30T06:46:30.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
117622025-04-30T06:46:30.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b235da08-27ac-4f15-98ee-e438ec0522c6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117632025-04-30T06:46:30.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
117642025-04-30T06:46:30.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a277a0b-fee6-4b45-8095-c089c2be72bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117652025-04-30T06:46:30.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
117662025-04-30T06:46:30.723ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04411c2c-34a0-4cf2-aed5-7e470b71f690", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117672025-04-30T06:46:30.723ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
117682025-04-30T06:46:30.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac9db4b-306b-487a-a3a3-b0c957add35f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117692025-04-30T06:46:30.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
117702025-04-30T06:46:30.726ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23967beb-2349-49ac-9277-ddf219e48229", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117712025-04-30T06:46:30.726ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
117722025-04-30T06:46:30.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ba03a8a-9f63-4bf9-9dc4-b124d80173ee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117732025-04-30T06:46:30.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
117742025-04-30T06:46:30.728ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bfeb359-269e-4f60-9f1d-0842bad102a0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
117752025-04-30T06:46:30.728ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117762025-04-30T06:46:30.728ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117772025-04-30T06:46:30.728ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117782025-04-30T06:46:30.728ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117792025-04-30T06:46:30.729ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117802025-04-30T06:46:30.734ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117812025-04-30T06:46:30.739ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117822025-04-30T06:46:30.740ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117832025-04-30T06:46:30.740ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117842025-04-30T06:46:30.740ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117852025-04-30T06:46:30.740ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117862025-04-30T06:46:30.741ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117872025-04-30T06:46:30.745ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117882025-04-30T06:46:30.750ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117892025-04-30T06:46:30.750ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117902025-04-30T06:46:30.750ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117912025-04-30T06:46:30.750ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117922025-04-30T06:46:30.751ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117932025-04-30T06:46:30.751ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117942025-04-30T06:46:30.751ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117952025-04-30T06:46:30.751ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
117962025-04-30T06:46:30.751ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117972025-04-30T06:46:30.798ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
117982025-04-30T06:46:30.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
117992025-04-30T06:46:30.799ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118002025-04-30T06:46:30.804ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118012025-04-30T06:46:30.805ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118022025-04-30T06:46:30.805ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
118032025-04-30T06:46:30.805ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118042025-04-30T06:46:30.805ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:53667 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
118052025-04-30T06:46:30.805ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:53667 req_id = da6d77ff-f8f8-47e9-9858-35441af86a34 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118062025-04-30T06:46:30.806ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 586 local_addr = [::1]:54590 method = GET remote_addr = [::1]:53667 req_id = da6d77ff-f8f8-47e9-9858-35441af86a34 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118072025-04-30T06:46:30.807ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "da6d77ff-f8f8-47e9-9858-35441af86a34", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118082025-04-30T06:46:30.808ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118092025-04-30T06:46:30.808ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118102025-04-30T06:46:30.808ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
118112025-04-30T06:46:30.810ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118122025-04-30T06:46:30.810ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118132025-04-30T06:46:30.810ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118142025-04-30T06:46:30.810ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118152025-04-30T06:46:30.810ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118162025-04-30T06:46:30.810ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118172025-04-30T06:46:30.810ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
118182025-04-30T06:46:30.811ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118192025-04-30T06:46:30.814ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118202025-04-30T06:46:30.827ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118212025-04-30T06:46:30.827ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118222025-04-30T06:46:30.827ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118232025-04-30T06:46:30.828ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118242025-04-30T06:46:30.867ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
118252025-04-30T06:46:30.870ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
118262025-04-30T06:46:30.870ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
118272025-04-30T06:46:30.870ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 472.007144ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
118282025-04-30T06:46:30.870ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
118292025-04-30T06:46:30.870ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118302025-04-30T06:46:30.870ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118312025-04-30T06:46:30.870ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118322025-04-30T06:46:30.870ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118332025-04-30T06:46:30.871ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118342025-04-30T06:46:30.874ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118352025-04-30T06:46:30.879ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118362025-04-30T06:46:30.879ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118372025-04-30T06:46:30.879ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118382025-04-30T06:46:30.880ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118392025-04-30T06:46:30.922ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118402025-04-30T06:46:30.923ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118412025-04-30T06:46:30.925ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118422025-04-30T06:46:30.925ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
118432025-04-30T06:46:30.926ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118442025-04-30T06:46:30.927ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
118452025-04-30T06:46:30.927ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
118462025-04-30T06:46:30.939ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118472025-04-30T06:46:30.946ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118482025-04-30T06:46:30.946ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118492025-04-30T06:46:30.946ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
118502025-04-30T06:46:30.947ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118512025-04-30T06:46:30.947ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fd 34 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .4..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53938 req_id = a0ee2a1e-107c-48ec-ba5d-b475a3dfc8f7
118522025-04-30T06:46:30.948ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64820,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53938 req_id = a0ee2a1e-107c-48ec-ba5d-b475a3dfc8f7
118532025-04-30T06:46:30.948ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
118542025-04-30T06:46:30.948ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118552025-04-30T06:46:30.948ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53938 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a0ee2a1e-107c-48ec-ba5d-b475a3dfc8f7
118562025-04-30T06:46:30.949ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4455, tv_nsec: 748101578 } })
118572025-04-30T06:46:30.949ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
118582025-04-30T06:46:30.949ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 62 35 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e b5..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49336 req_id = c253ab17-730a-4bfb-a3c1-fb01750b7ab3
118592025-04-30T06:46:30.949ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25141,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49336 req_id = c253ab17-730a-4bfb-a3c1-fb01750b7ab3
118602025-04-30T06:46:30.950ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118612025-04-30T06:46:30.950ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49336 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c253ab17-730a-4bfb-a3c1-fb01750b7ab3
118622025-04-30T06:46:31.007ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118632025-04-30T06:46:31.008ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118642025-04-30T06:46:31.008ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118652025-04-30T06:46:31.009ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
118662025-04-30T06:46:31.010ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
118672025-04-30T06:46:31.011ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118682025-04-30T06:46:31.011ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
118692025-04-30T06:46:31.012ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118702025-04-30T06:46:31.015ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 57 57 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c WW..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59093 req_id = cf88b10f-6c13-42cd-8800-1ff0c03769a8
118712025-04-30T06:46:31.015ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22359,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59093 req_id = cf88b10f-6c13-42cd-8800-1ff0c03769a8
118722025-04-30T06:46:31.015ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
118732025-04-30T06:46:31.015ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22359, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59093 req_id = cf88b10f-6c13-42cd-8800-1ff0c03769a8
118742025-04-30T06:46:31.016ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
118752025-04-30T06:46:31.016ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
118762025-04-30T06:46:31.016ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
118772025-04-30T06:46:31.016ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
118782025-04-30T06:46:31.016ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
118792025-04-30T06:46:31.022ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
118802025-04-30T06:46:31.023ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118812025-04-30T06:46:31.026ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "be214dad-07e5-42d2-8b53-a3e0f9a60d43", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118822025-04-30T06:46:31.026ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
118832025-04-30T06:46:31.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9012d49-9135-482a-a36a-113f85ef0ca9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118842025-04-30T06:46:31.029ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
118852025-04-30T06:46:31.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06ae0844-8318-4adf-8c10-8fc3d997178d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118862025-04-30T06:46:31.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
118872025-04-30T06:46:31.056ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3eace12-ccdd-4bba-b649-8e7611ab1f40", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118882025-04-30T06:46:31.056ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
118892025-04-30T06:46:31.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dd058ea-a1c9-4fd6-a7be-d2a4a11917aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118902025-04-30T06:46:31.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
118912025-04-30T06:46:31.063ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
118922025-04-30T06:46:31.068ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "741af856-4004-4411-b4c9-3a8594895c96", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118932025-04-30T06:46:31.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
118942025-04-30T06:46:31.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda07d1b-d7c4-46bf-9f57-d1403bf3e2ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118952025-04-30T06:46:31.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
118962025-04-30T06:46:31.075ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "861b9a4d-3ab8-478b-b47a-0858e5e0c92b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118972025-04-30T06:46:31.075ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
118982025-04-30T06:46:31.077ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "277aad5b-c730-4d6d-9741-d69959be2843", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
118992025-04-30T06:46:31.077ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
119002025-04-30T06:46:31.078ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d8a88b-c6f0-483d-9c1a-b091fc0e91cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119012025-04-30T06:46:31.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
119022025-04-30T06:46:31.080ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efc49833-d6fc-45ed-ab85-c1cb17d34c83", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119032025-04-30T06:46:31.080ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
119042025-04-30T06:46:31.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1032c41f-344b-476a-b34c-44ec4b1d18fa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119052025-04-30T06:46:31.084ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
119062025-04-30T06:46:31.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6caf13c-48dc-4e15-82ae-6823b12918ad", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119072025-04-30T06:46:31.085ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
119082025-04-30T06:46:31.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed83ac11-69c4-4b63-9afe-98415ccd779f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119092025-04-30T06:46:31.093ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
119102025-04-30T06:46:31.109ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e5535b-1ca9-4482-be1c-e1e7c84df84f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119112025-04-30T06:46:31.110ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
119122025-04-30T06:46:31.111ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
119132025-04-30T06:46:31.116ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119142025-04-30T06:46:31.116ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119152025-04-30T06:46:31.116ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119162025-04-30T06:46:31.116ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119172025-04-30T06:46:31.117ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119182025-04-30T06:46:31.121ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119192025-04-30T06:46:31.127ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119202025-04-30T06:46:31.127ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119212025-04-30T06:46:31.127ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119222025-04-30T06:46:31.127ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119232025-04-30T06:46:31.130ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119242025-04-30T06:46:31.130ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119252025-04-30T06:46:31.131ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119262025-04-30T06:46:31.133ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6afe51b-e773-429e-8d8a-73d29bb66b36", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119272025-04-30T06:46:31.133ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
119282025-04-30T06:46:31.138ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4e078e-117b-4720-9216-97e50751fe6a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119292025-04-30T06:46:31.138ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
119302025-04-30T06:46:31.143ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a88dfbf5-7260-4023-af16-ec65f2954a0e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119312025-04-30T06:46:31.143ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
119322025-04-30T06:46:31.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aed1e15-e6ec-493e-95d9-a01beac09c6c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119332025-04-30T06:46:31.150ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
119342025-04-30T06:46:31.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f7ca0f5-e254-434a-887c-0062fe223682", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119352025-04-30T06:46:31.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
119362025-04-30T06:46:31.155ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8523384-eb7b-4938-8069-397f9767538d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119372025-04-30T06:46:31.155ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
119382025-04-30T06:46:31.158ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1796b8-3881-4c41-8d71-b1da15edd7eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119392025-04-30T06:46:31.158ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
119402025-04-30T06:46:31.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a93b3c9-d4f3-4d44-b018-1fa0bdd19bf2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119412025-04-30T06:46:31.160ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
119422025-04-30T06:46:31.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a0f566-cc24-4e5e-94bd-e779d4433992", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119432025-04-30T06:46:31.163ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
119442025-04-30T06:46:31.164ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
119452025-04-30T06:46:31.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5442276e-f5ae-49d1-b035-7c2d2d3de542", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119462025-04-30T06:46:31.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
119472025-04-30T06:46:31.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66db444e-76b7-4241-a062-5517db241fa0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119482025-04-30T06:46:31.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
119492025-04-30T06:46:31.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d072d8f5-cc9d-4111-87bf-2dabb89c957e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119502025-04-30T06:46:31.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
119512025-04-30T06:46:31.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1fd9470-311e-4332-9691-4f83666d6514", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119522025-04-30T06:46:31.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
119532025-04-30T06:46:31.179ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4c1ee10-a291-4782-8049-4a3625aaa17f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119542025-04-30T06:46:31.179ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
119552025-04-30T06:46:31.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad80f95d-1a6b-42e0-979e-b8aa23badcb2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119562025-04-30T06:46:31.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
119572025-04-30T06:46:31.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6287642-80fe-4812-8bab-ef62e7354e7c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119582025-04-30T06:46:31.182ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
119592025-04-30T06:46:31.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f80a466e-20d1-49f4-bd1b-aa61d813fe22", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119602025-04-30T06:46:31.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
119612025-04-30T06:46:31.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "779ae0d8-cf18-4da0-9684-d6764d7aaf28", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119622025-04-30T06:46:31.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
119632025-04-30T06:46:31.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4f8cf6e-2a4a-498e-9395-be92904c9089", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119642025-04-30T06:46:31.186ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
119652025-04-30T06:46:31.192ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5e54331-cb10-48ba-8799-c7caf0c8e31b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119662025-04-30T06:46:31.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
119672025-04-30T06:46:31.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd4e7648-8862-4b04-9c3b-0594d773b91d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119682025-04-30T06:46:31.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
119692025-04-30T06:46:31.200ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "787b61a4-fd9c-40ee-bb2b-5ed2447189f8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119702025-04-30T06:46:31.200ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
119712025-04-30T06:46:31.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1efe5f9-260f-4a93-baf5-773bda95b33f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119722025-04-30T06:46:31.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
119732025-04-30T06:46:31.215ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119742025-04-30T06:46:31.215ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119752025-04-30T06:46:31.216ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119762025-04-30T06:46:31.216ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119772025-04-30T06:46:31.216ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119782025-04-30T06:46:31.218ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119792025-04-30T06:46:31.224ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119802025-04-30T06:46:31.224ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119812025-04-30T06:46:31.224ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119822025-04-30T06:46:31.224ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119832025-04-30T06:46:31.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf5ecabe-cc58-4803-86cc-37a2dc972c53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119842025-04-30T06:46:31.225ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
119852025-04-30T06:46:31.227ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119862025-04-30T06:46:31.227ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
119872025-04-30T06:46:31.227ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119882025-04-30T06:46:31.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad844e64-c2d8-45b5-8a3d-44b1c32336c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119892025-04-30T06:46:31.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
119902025-04-30T06:46:31.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d246b6f-39fd-4e66-9a99-d2c7a8c65b47", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119912025-04-30T06:46:31.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
119922025-04-30T06:46:31.239ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61d38cab-dca5-4d24-8799-93a93fd596e9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119932025-04-30T06:46:31.239ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
119942025-04-30T06:46:31.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "966be8e5-42d1-4ef2-9ee7-d25cc6b807b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119952025-04-30T06:46:31.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
119962025-04-30T06:46:31.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f4658d7-8752-4ce0-a1df-ef109537dbc9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119972025-04-30T06:46:31.241ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
119982025-04-30T06:46:31.243ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68c9134b-7db0-49f8-afda-a947841d47cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
119992025-04-30T06:46:31.243ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120002025-04-30T06:46:31.243ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120012025-04-30T06:46:31.243ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120022025-04-30T06:46:31.243ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120032025-04-30T06:46:31.243ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120042025-04-30T06:46:31.247ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120052025-04-30T06:46:31.252ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120062025-04-30T06:46:31.252ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120072025-04-30T06:46:31.252ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120082025-04-30T06:46:31.252ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120092025-04-30T06:46:31.252ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120102025-04-30T06:46:31.253ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120112025-04-30T06:46:31.283ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120122025-04-30T06:46:31.289ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120132025-04-30T06:46:31.289ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120142025-04-30T06:46:31.289ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120152025-04-30T06:46:31.289ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120162025-04-30T06:46:31.354ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
120172025-04-30T06:46:31.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
120182025-04-30T06:46:31.355ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
120192025-04-30T06:46:31.355ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:58088 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
120202025-04-30T06:46:31.356ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:58088 req_id = ef3419c1-ea80-4f82-b464-e00d3f4e269c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120212025-04-30T06:46:31.356ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 584 local_addr = [::1]:54590 method = GET remote_addr = [::1]:58088 req_id = ef3419c1-ea80-4f82-b464-e00d3f4e269c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120222025-04-30T06:46:31.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef3419c1-ea80-4f82-b464-e00d3f4e269c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
120232025-04-30T06:46:31.359ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120242025-04-30T06:46:31.360ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120252025-04-30T06:46:31.360ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
120262025-04-30T06:46:31.363ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120272025-04-30T06:46:31.363ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120282025-04-30T06:46:31.363ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120292025-04-30T06:46:31.363ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120302025-04-30T06:46:31.365ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120312025-04-30T06:46:31.369ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120322025-04-30T06:46:31.375ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120332025-04-30T06:46:31.375ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120342025-04-30T06:46:31.375ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120352025-04-30T06:46:31.375ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120362025-04-30T06:46:31.406ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120372025-04-30T06:46:31.406ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120382025-04-30T06:46:31.408ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120392025-04-30T06:46:31.408ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
120402025-04-30T06:46:31.409ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120412025-04-30T06:46:31.412ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120422025-04-30T06:46:31.413ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120432025-04-30T06:46:31.413ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 542.974451ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120442025-04-30T06:46:31.413ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
120452025-04-30T06:46:31.413ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120462025-04-30T06:46:31.413ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120472025-04-30T06:46:31.413ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120482025-04-30T06:46:31.413ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120492025-04-30T06:46:31.414ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120502025-04-30T06:46:31.416ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120512025-04-30T06:46:31.421ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120522025-04-30T06:46:31.421ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120532025-04-30T06:46:31.421ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120542025-04-30T06:46:31.422ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120552025-04-30T06:46:31.422ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120562025-04-30T06:46:31.430ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120572025-04-30T06:46:31.430ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120582025-04-30T06:46:31.430ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
120592025-04-30T06:46:31.431ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120602025-04-30T06:46:31.448ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
120612025-04-30T06:46:31.448ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120622025-04-30T06:46:31.448ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120632025-04-30T06:46:31.448ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
120642025-04-30T06:46:31.450ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120652025-04-30T06:46:31.453ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 b2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52188 req_id = e1d5448f-ac1e-49ec-956e-c0e6dd005148
120662025-04-30T06:46:31.453ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45746,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52188 req_id = e1d5448f-ac1e-49ec-956e-c0e6dd005148
120672025-04-30T06:46:31.456ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
120682025-04-30T06:46:31.456ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120692025-04-30T06:46:31.456ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52188 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e1d5448f-ac1e-49ec-956e-c0e6dd005148
120702025-04-30T06:46:31.456ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4456, tv_nsec: 255498424 } })
120712025-04-30T06:46:31.456ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
120722025-04-30T06:46:31.459ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bc 0a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60803 req_id = 1737b5e0-92f0-43c5-ac9c-19052449edcf
120732025-04-30T06:46:31.459ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48138,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60803 req_id = 1737b5e0-92f0-43c5-ac9c-19052449edcf
120742025-04-30T06:46:31.459ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120752025-04-30T06:46:31.459ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60803 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1737b5e0-92f0-43c5-ac9c-19052449edcf
120762025-04-30T06:46:31.519ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120772025-04-30T06:46:31.530ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 79 31 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c y1..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54518 req_id = 02a19d97-7711-4c96-aa28-3d4cb31bd95c
120782025-04-30T06:46:31.531ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31025,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54518 req_id = 02a19d97-7711-4c96-aa28-3d4cb31bd95c
120792025-04-30T06:46:31.531ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
120802025-04-30T06:46:31.531ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31025, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54518 req_id = 02a19d97-7711-4c96-aa28-3d4cb31bd95c
120812025-04-30T06:46:31.532ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
120822025-04-30T06:46:31.532ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
120832025-04-30T06:46:31.532ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
120842025-04-30T06:46:31.539ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "37eab473-302f-46e4-b08e-fb01c1e01e8c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
120852025-04-30T06:46:31.540ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
120862025-04-30T06:46:31.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf7c5fc9-d3d2-4f3f-8015-2b83587d83ec", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
120872025-04-30T06:46:31.543ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
120882025-04-30T06:46:31.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9faf30a2-93b8-4555-bc8c-32869f1b0a90", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
120892025-04-30T06:46:31.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
120902025-04-30T06:46:31.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
120912025-04-30T06:46:31.549ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120922025-04-30T06:46:31.550ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 5408b0a8-33aa-496a-990a-98a16e584d27 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120932025-04-30T06:46:31.551ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 5408b0a8-33aa-496a-990a-98a16e584d27 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120942025-04-30T06:46:31.551ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 5408b0a8-33aa-496a-990a-98a16e584d27 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120952025-04-30T06:46:31.551ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 5408b0a8-33aa-496a-990a-98a16e584d27 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120962025-04-30T06:46:31.555ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 5109 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 5408b0a8-33aa-496a-990a-98a16e584d27 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120972025-04-30T06:46:31.557ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5408b0a8-33aa-496a-990a-98a16e584d27", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
120982025-04-30T06:46:31.557ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120992025-04-30T06:46:31.563ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 59063b02-1fbc-4a39-9d8d-456021bec18c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
121002025-04-30T06:46:31.563ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 59063b02-1fbc-4a39-9d8d-456021bec18c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
121012025-04-30T06:46:31.564ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 59063b02-1fbc-4a39-9d8d-456021bec18c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
121022025-04-30T06:46:31.565ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 59063b02-1fbc-4a39-9d8d-456021bec18c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
121032025-04-30T06:46:31.565ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
121042025-04-30T06:46:31.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4e0a681-0ec6-4175-a772-8f5277c6b7e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121052025-04-30T06:46:31.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
121062025-04-30T06:46:31.569ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 5503 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 59063b02-1fbc-4a39-9d8d-456021bec18c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
121072025-04-30T06:46:31.569ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "59063b02-1fbc-4a39-9d8d-456021bec18c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121082025-04-30T06:46:31.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
121092025-04-30T06:46:31.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121102025-04-30T06:46:31.569ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
121112025-04-30T06:46:31.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f786eb59-61f5-4ea0-8e09-32d4089562de", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121122025-04-30T06:46:31.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
121132025-04-30T06:46:31.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5ad6292-ea27-4824-b381-a05443b7d0d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121142025-04-30T06:46:31.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
121152025-04-30T06:46:31.577ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69297e13-9200-4c11-afd0-efcba7db5614", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121162025-04-30T06:46:31.577ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
121172025-04-30T06:46:31.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6915f36-cd25-4cc2-95ec-03b74813c23b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121182025-04-30T06:46:31.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
121192025-04-30T06:46:31.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "532c7256-5113-41c5-84f3-ee703b7d80b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121202025-04-30T06:46:31.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
121212025-04-30T06:46:31.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74265d77-fabe-434d-aa73-ed7e7231a30e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121222025-04-30T06:46:31.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
121232025-04-30T06:46:31.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cdf911f-20a2-46eb-8534-0ff3ec61e64c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121242025-04-30T06:46:31.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
121252025-04-30T06:46:31.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa21c61-0dbe-426a-bede-8f4f50b44673", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121262025-04-30T06:46:31.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
121272025-04-30T06:46:31.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d869e3-dd89-4c18-b804-ca3bd6817941", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121282025-04-30T06:46:31.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
121292025-04-30T06:46:31.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd32a9d1-4051-4b79-bc6e-66e61f6d2334", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121302025-04-30T06:46:31.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
121312025-04-30T06:46:31.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa896d81-14b3-4793-889d-789c1ac76618", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121322025-04-30T06:46:31.593ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
121332025-04-30T06:46:31.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0281061-8bfd-49a9-9976-bcef6b005f38", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121342025-04-30T06:46:31.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
121352025-04-30T06:46:31.597ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3447bf58-b300-4364-8357-94c747fb4cd1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121362025-04-30T06:46:31.597ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
121372025-04-30T06:46:31.601ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfdce64a-4c11-465d-b2eb-d7098b5f4b12", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121382025-04-30T06:46:31.601ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
121392025-04-30T06:46:31.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25982b3c-a9a6-41c5-9b47-32420f97bc3f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121402025-04-30T06:46:31.603ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
121412025-04-30T06:46:31.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ad4112f-0639-4c79-bf20-3391a5ef9efc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121422025-04-30T06:46:31.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
121432025-04-30T06:46:31.606ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3fbe9b-943e-4b0c-879c-883e901b5e94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121442025-04-30T06:46:31.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
121452025-04-30T06:46:31.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64d00003-575f-4e6e-bc9f-d2c544833d91", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121462025-04-30T06:46:31.611ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
121472025-04-30T06:46:31.615ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5268d838-e8bd-41b3-b438-ef926db4a699", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121482025-04-30T06:46:31.615ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
121492025-04-30T06:46:31.616ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121502025-04-30T06:46:31.616ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121512025-04-30T06:46:31.616ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121522025-04-30T06:46:31.616ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
121532025-04-30T06:46:31.618ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121542025-04-30T06:46:31.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "feca8d08-2ceb-44a6-a61a-d0e6c93680ac", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121552025-04-30T06:46:31.619ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
121562025-04-30T06:46:31.628ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121572025-04-30T06:46:31.643ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121582025-04-30T06:46:31.643ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121592025-04-30T06:46:31.643ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
121602025-04-30T06:46:31.645ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121612025-04-30T06:46:31.646ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121622025-04-30T06:46:31.646ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121632025-04-30T06:46:31.648ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121642025-04-30T06:46:31.648ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
121652025-04-30T06:46:31.649ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121662025-04-30T06:46:31.650ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121672025-04-30T06:46:31.652ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121682025-04-30T06:46:31.652ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121692025-04-30T06:46:31.652ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121702025-04-30T06:46:31.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f48e0555-7a45-45fe-b245-5e29e527702e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121712025-04-30T06:46:31.654ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
121722025-04-30T06:46:31.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db2bacba-2bfb-4675-9bef-d5a1846b5c6f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:30 GMT"} })
121732025-04-30T06:46:31.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
121742025-04-30T06:46:31.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e53ebfc0-016f-4739-aa56-0629c1b51b48", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121752025-04-30T06:46:31.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
121762025-04-30T06:46:31.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "629d5154-e5a4-4e54-84bc-0d11369924cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121772025-04-30T06:46:31.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
121782025-04-30T06:46:31.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d028ce3-0e03-454a-b971-64990477f169", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121792025-04-30T06:46:31.690ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
121802025-04-30T06:46:31.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d392df7-ed0d-4f80-9fc0-44ecc5efc33e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121812025-04-30T06:46:31.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
121822025-04-30T06:46:31.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "032bcafd-1c2b-4468-95fe-7b07e2513d52", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121832025-04-30T06:46:31.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
121842025-04-30T06:46:31.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10edf9c0-1746-4941-b6e2-a6c6ef8ca524", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121852025-04-30T06:46:31.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
121862025-04-30T06:46:31.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4acc6799-0369-4a90-bb29-a30d2b85487e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121872025-04-30T06:46:31.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
121882025-04-30T06:46:31.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3aba5bb-2da1-4714-a634-cef1cd86dae8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121892025-04-30T06:46:31.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
121902025-04-30T06:46:31.703ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
121912025-04-30T06:46:31.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10ce56c4-05b2-400b-9778-1f8b2775e8d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121922025-04-30T06:46:31.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
121932025-04-30T06:46:31.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c65f991d-62e3-4d9f-afe6-5a6145fda188", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121942025-04-30T06:46:31.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
121952025-04-30T06:46:31.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9b4365-c0e8-4e99-9d59-99625542e8fa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121962025-04-30T06:46:31.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
121972025-04-30T06:46:31.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a802aabf-6a9e-4906-8847-1b76d7cbb3e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
121982025-04-30T06:46:31.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
121992025-04-30T06:46:31.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f303e900-3d25-4147-9831-addd02905d78", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122002025-04-30T06:46:31.725ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
122012025-04-30T06:46:31.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ee5f99b-9283-4cd6-ab42-ec7e37e11578", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122022025-04-30T06:46:31.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
122032025-04-30T06:46:31.729ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe36c5e3-af4f-4f6f-9f16-0d2148c62590", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122042025-04-30T06:46:31.730ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
122052025-04-30T06:46:31.731ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab4bf9c-276b-441d-9997-a896266e2fd0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122062025-04-30T06:46:31.731ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
122072025-04-30T06:46:31.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f215063d-9ac6-4536-a019-342ffaa6ec31", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122082025-04-30T06:46:31.733ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
122092025-04-30T06:46:31.735ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c1f7b1d-a1fd-45f8-a6a7-59f09a3a0425", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122102025-04-30T06:46:31.735ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
122112025-04-30T06:46:31.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f4f22b-5837-49f7-a5a1-5419be4fea34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122122025-04-30T06:46:31.737ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122132025-04-30T06:46:31.737ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122142025-04-30T06:46:31.737ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122152025-04-30T06:46:31.737ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122162025-04-30T06:46:31.738ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122172025-04-30T06:46:31.741ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122182025-04-30T06:46:31.745ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122192025-04-30T06:46:31.745ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122202025-04-30T06:46:31.745ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122212025-04-30T06:46:31.745ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122222025-04-30T06:46:31.745ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122232025-04-30T06:46:31.745ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122242025-04-30T06:46:31.748ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122252025-04-30T06:46:31.753ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122262025-04-30T06:46:31.753ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122272025-04-30T06:46:31.753ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122282025-04-30T06:46:31.754ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122292025-04-30T06:46:31.802ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
122302025-04-30T06:46:31.802ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
122312025-04-30T06:46:31.803ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122322025-04-30T06:46:31.803ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122332025-04-30T06:46:31.803ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122342025-04-30T06:46:31.803ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
122352025-04-30T06:46:31.804ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122362025-04-30T06:46:31.804ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:38730 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122372025-04-30T06:46:31.805ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:38730 req_id = 241bf66f-648b-4dc7-aea3-df495ba157a3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122382025-04-30T06:46:31.805ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 700 local_addr = [::1]:54590 method = GET remote_addr = [::1]:38730 req_id = 241bf66f-648b-4dc7-aea3-df495ba157a3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122392025-04-30T06:46:31.806ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "241bf66f-648b-4dc7-aea3-df495ba157a3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
122402025-04-30T06:46:31.807ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122412025-04-30T06:46:31.807ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122422025-04-30T06:46:31.807ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
122432025-04-30T06:46:31.808ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122442025-04-30T06:46:31.808ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122452025-04-30T06:46:31.808ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122462025-04-30T06:46:31.808ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122472025-04-30T06:46:31.809ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122482025-04-30T06:46:31.810ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122492025-04-30T06:46:31.819ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122502025-04-30T06:46:31.819ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122512025-04-30T06:46:31.819ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
122522025-04-30T06:46:31.820ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122532025-04-30T06:46:31.822ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122542025-04-30T06:46:31.826ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122552025-04-30T06:46:31.826ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122562025-04-30T06:46:31.826ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122572025-04-30T06:46:31.827ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122582025-04-30T06:46:31.828ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122592025-04-30T06:46:31.828ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
122602025-04-30T06:46:31.829ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122612025-04-30T06:46:31.904ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122622025-04-30T06:46:31.905ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122632025-04-30T06:46:31.905ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 491.757424ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122642025-04-30T06:46:31.905ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
122652025-04-30T06:46:31.905ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122662025-04-30T06:46:31.905ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122672025-04-30T06:46:31.905ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122682025-04-30T06:46:31.905ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122692025-04-30T06:46:31.910ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122702025-04-30T06:46:31.917ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122712025-04-30T06:46:31.927ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122722025-04-30T06:46:31.927ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122732025-04-30T06:46:31.927ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122742025-04-30T06:46:31.927ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122752025-04-30T06:46:31.928ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
122762025-04-30T06:46:31.932ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
122772025-04-30T06:46:31.932ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122782025-04-30T06:46:31.932ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d b1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60507 req_id = b2bdfa34-a4ff-479f-a859-0b45c4c26053
122792025-04-30T06:46:31.933ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3505,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60507 req_id = b2bdfa34-a4ff-479f-a859-0b45c4c26053
122802025-04-30T06:46:31.933ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
122812025-04-30T06:46:31.934ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122822025-04-30T06:46:31.934ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60507 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2bdfa34-a4ff-479f-a859-0b45c4c26053
122832025-04-30T06:46:31.934ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4456, tv_nsec: 733359390 } })
122842025-04-30T06:46:31.934ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
122852025-04-30T06:46:31.934ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a5 d1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64805 req_id = 3dce75cd-8362-4f76-b704-dc00789108d8
122862025-04-30T06:46:31.934ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42449,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64805 req_id = 3dce75cd-8362-4f76-b704-dc00789108d8
122872025-04-30T06:46:31.935ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122882025-04-30T06:46:31.935ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64805 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3dce75cd-8362-4f76-b704-dc00789108d8
122892025-04-30T06:46:31.989ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122902025-04-30T06:46:31.990ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122912025-04-30T06:46:31.990ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122922025-04-30T06:46:31.990ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122932025-04-30T06:46:31.990ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
122942025-04-30T06:46:31.990ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122952025-04-30T06:46:31.991ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122962025-04-30T06:46:31.991ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122972025-04-30T06:46:31.991ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
122982025-04-30T06:46:31.992ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
122992025-04-30T06:46:31.993ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 43 38 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c C8..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52810 req_id = 446dd138-95ca-491a-94f6-2666447cdf49
123002025-04-30T06:46:31.993ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17208,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52810 req_id = 446dd138-95ca-491a-94f6-2666447cdf49
123012025-04-30T06:46:31.993ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
123022025-04-30T06:46:31.993ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17208, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52810 req_id = 446dd138-95ca-491a-94f6-2666447cdf49
123032025-04-30T06:46:31.993ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
123042025-04-30T06:46:31.993ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
123052025-04-30T06:46:31.993ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
123062025-04-30T06:46:31.995ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123072025-04-30T06:46:32.001ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123082025-04-30T06:46:32.002ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123092025-04-30T06:46:32.002ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
123102025-04-30T06:46:32.002ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123112025-04-30T06:46:32.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "83a9970f-3ff0-4efc-bf63-07a8d084eb39", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123122025-04-30T06:46:32.003ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
123132025-04-30T06:46:32.003ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
123142025-04-30T06:46:32.003ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
123152025-04-30T06:46:32.005ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123162025-04-30T06:46:32.005ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
123172025-04-30T06:46:32.006ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123182025-04-30T06:46:32.006ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b94ba7f2-94f8-4c38-b41c-38b5d6a9fb60", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123192025-04-30T06:46:32.007ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
123202025-04-30T06:46:32.007ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
123212025-04-30T06:46:32.007ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123222025-04-30T06:46:32.012ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9b8edab-61ac-45c6-84b9-6219990b776a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123232025-04-30T06:46:32.012ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
123242025-04-30T06:46:32.016ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2603364c-5d38-421e-b672-3a31594dee8b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123252025-04-30T06:46:32.017ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
123262025-04-30T06:46:32.021ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0a9a77-f007-47cd-86b9-78a18304e42e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123272025-04-30T06:46:32.021ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
123282025-04-30T06:46:32.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff3f8d9-2ce9-4939-9af7-64f254783b51", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123292025-04-30T06:46:32.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
123302025-04-30T06:46:32.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c092e54-566e-448f-98dd-99fde914bbc3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123312025-04-30T06:46:32.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
123322025-04-30T06:46:32.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "347a60e9-9b5a-49e1-af5d-998a51f314cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123332025-04-30T06:46:32.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
123342025-04-30T06:46:32.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22e72a60-5eb1-4bc7-8a63-f1ca6cf7dca8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123352025-04-30T06:46:32.033ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
123362025-04-30T06:46:32.034ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03bbc97c-f4f8-4b22-8a08-52f976182bc0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123372025-04-30T06:46:32.034ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
123382025-04-30T06:46:32.037ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f489ed52-ff5f-48a8-80ce-2f320886a225", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123392025-04-30T06:46:32.037ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
123402025-04-30T06:46:32.040ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "950ac20a-7565-41fa-a040-de8cd2d67a14", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123412025-04-30T06:46:32.040ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
123422025-04-30T06:46:32.041ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
123432025-04-30T06:46:32.048ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a424cb2-d985-413b-804f-53d7d15dbdb1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123442025-04-30T06:46:32.049ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
123452025-04-30T06:46:32.051ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fbaa314-a5e6-435a-9394-5535bb5d7b71", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123462025-04-30T06:46:32.051ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
123472025-04-30T06:46:32.053ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e25d231-e089-404c-97a6-307f0df430e8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123482025-04-30T06:46:32.053ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
123492025-04-30T06:46:32.054ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74c2c85c-913c-4fe8-903b-3f801b836d6b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123502025-04-30T06:46:32.054ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
123512025-04-30T06:46:32.056ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98538574-a679-4979-ad77-fb73d45629f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123522025-04-30T06:46:32.057ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
123532025-04-30T06:46:32.058ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09a65820-2840-4626-bc5e-6d5a9ad716af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123542025-04-30T06:46:32.058ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
123552025-04-30T06:46:32.060ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a5f1c5e-2828-481f-9b44-e6e47d817bd8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123562025-04-30T06:46:32.060ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
123572025-04-30T06:46:32.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7637af48-41d1-49fb-bc44-10566c9aa041", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123582025-04-30T06:46:32.061ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
123592025-04-30T06:46:32.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a4e42a-0c7e-4ff7-b874-bbc451fae8f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123602025-04-30T06:46:32.062ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
123612025-04-30T06:46:32.063ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34942653-7a77-49bf-ba73-2d9df513ade6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123622025-04-30T06:46:32.063ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
123632025-04-30T06:46:32.064ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1af0a1e2-6a3e-4f5c-a1e6-1ba3e9f0534e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123642025-04-30T06:46:32.064ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
123652025-04-30T06:46:32.066ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77cdc1eb-6c42-4e5a-b90f-01f741539910", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123662025-04-30T06:46:32.066ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
123672025-04-30T06:46:32.067ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c320a679-b8e1-4328-b849-b35e7245dbd2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123682025-04-30T06:46:32.068ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
123692025-04-30T06:46:32.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0012ea-b2ba-4da3-ad1d-5d380e562dde", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123702025-04-30T06:46:32.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
123712025-04-30T06:46:32.071ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1d1619e-550e-4fc7-8e2d-fbfabe8e5954", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123722025-04-30T06:46:32.072ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
123732025-04-30T06:46:32.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "175ffcd6-16d5-4390-9998-ee4e12cfdab7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123742025-04-30T06:46:32.074ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
123752025-04-30T06:46:32.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c528cd8c-c0a9-44c0-b1e9-731d94921290", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123762025-04-30T06:46:32.076ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
123772025-04-30T06:46:32.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcc93c6f-2bfe-40ff-923f-eb1a6409806f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123782025-04-30T06:46:32.079ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
123792025-04-30T06:46:32.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d35b834-ac9a-4e88-b0e6-6432ca785bf8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123802025-04-30T06:46:32.088ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
123812025-04-30T06:46:32.089ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ae9a2be-addd-46af-928e-dd30500ecd60", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123822025-04-30T06:46:32.089ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
123832025-04-30T06:46:32.090ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0c3bd7-5b6c-4a1f-b4c0-003e8aa4a94f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123842025-04-30T06:46:32.090ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
123852025-04-30T06:46:32.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b40b48-d999-4ba9-a1ed-0eed7f4702bd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123862025-04-30T06:46:32.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
123872025-04-30T06:46:32.091ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123882025-04-30T06:46:32.091ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123892025-04-30T06:46:32.091ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123902025-04-30T06:46:32.091ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
123912025-04-30T06:46:32.093ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123922025-04-30T06:46:32.093ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
123932025-04-30T06:46:32.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ab264c0-6fcc-4cca-afa6-e33082fc3f9e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
123942025-04-30T06:46:32.095ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
123952025-04-30T06:46:32.095ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123962025-04-30T06:46:32.101ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123972025-04-30T06:46:32.101ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123982025-04-30T06:46:32.101ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
123992025-04-30T06:46:32.102ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124002025-04-30T06:46:32.104ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124012025-04-30T06:46:32.104ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
124022025-04-30T06:46:32.104ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124032025-04-30T06:46:32.105ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f4d66a-0fa8-42ec-8fa4-30c0d1377c8e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124042025-04-30T06:46:32.105ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
124052025-04-30T06:46:32.108ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd53519-b631-427b-976d-6eba4e5728d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124062025-04-30T06:46:32.108ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
124072025-04-30T06:46:32.111ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b81ea53-ff0d-4c3c-8bc4-8cd5f482155e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124082025-04-30T06:46:32.111ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
124092025-04-30T06:46:32.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "501a46b5-e206-4bae-bfe7-21d76ed277a2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124102025-04-30T06:46:32.126ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
124112025-04-30T06:46:32.129ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3304fa2c-cc1b-49fb-a816-e4eb1e3eb426", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124122025-04-30T06:46:32.129ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
124132025-04-30T06:46:32.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e885bb80-ace6-478e-9458-84e590ecc6f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124142025-04-30T06:46:32.134ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
124152025-04-30T06:46:32.137ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0058ba23-405a-4e7c-a157-7d55454ee2ee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124162025-04-30T06:46:32.137ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
124172025-04-30T06:46:32.138ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f145e9c8-e2c7-40e6-8bc9-4a8e354db32e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124182025-04-30T06:46:32.139ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
124192025-04-30T06:46:32.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd060221-15a9-4b4d-8e2c-e82207f3e7db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124202025-04-30T06:46:32.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
124212025-04-30T06:46:32.144ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aad6d97a-b1a6-48e4-a903-4b6486b5c800", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124222025-04-30T06:46:32.144ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124232025-04-30T06:46:32.144ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124242025-04-30T06:46:32.144ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124252025-04-30T06:46:32.144ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124262025-04-30T06:46:32.145ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124272025-04-30T06:46:32.146ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
124282025-04-30T06:46:32.151ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124292025-04-30T06:46:32.157ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124302025-04-30T06:46:32.157ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124312025-04-30T06:46:32.157ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124322025-04-30T06:46:32.157ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124332025-04-30T06:46:32.157ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124342025-04-30T06:46:32.158ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124352025-04-30T06:46:32.175ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124362025-04-30T06:46:32.179ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124372025-04-30T06:46:32.179ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124382025-04-30T06:46:32.179ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124392025-04-30T06:46:32.180ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124402025-04-30T06:46:32.239ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
124412025-04-30T06:46:32.240ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
124422025-04-30T06:46:32.243ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124432025-04-30T06:46:32.243ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124442025-04-30T06:46:32.243ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124452025-04-30T06:46:32.243ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
124462025-04-30T06:46:32.245ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124472025-04-30T06:46:32.246ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:34869 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124482025-04-30T06:46:32.248ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:34869 req_id = 171c3d44-acb6-4fa3-a3c2-04e7e07f2b4a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124492025-04-30T06:46:32.249ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 716 local_addr = [::1]:54590 method = GET remote_addr = [::1]:34869 req_id = 171c3d44-acb6-4fa3-a3c2-04e7e07f2b4a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124502025-04-30T06:46:32.249ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124512025-04-30T06:46:32.257ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124522025-04-30T06:46:32.257ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124532025-04-30T06:46:32.257ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
124542025-04-30T06:46:32.260ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124552025-04-30T06:46:32.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "171c3d44-acb6-4fa3-a3c2-04e7e07f2b4a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
124562025-04-30T06:46:32.262ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124572025-04-30T06:46:32.262ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124582025-04-30T06:46:32.262ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
124592025-04-30T06:46:32.264ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124602025-04-30T06:46:32.264ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124612025-04-30T06:46:32.264ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124622025-04-30T06:46:32.264ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124632025-04-30T06:46:32.264ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124642025-04-30T06:46:32.268ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124652025-04-30T06:46:32.268ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
124662025-04-30T06:46:32.269ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124672025-04-30T06:46:32.269ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124682025-04-30T06:46:32.274ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124692025-04-30T06:46:32.274ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124702025-04-30T06:46:32.274ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124712025-04-30T06:46:32.275ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124722025-04-30T06:46:32.308ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
124732025-04-30T06:46:32.308ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
124742025-04-30T06:46:32.309ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 403.628003ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
124752025-04-30T06:46:32.309ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
124762025-04-30T06:46:32.309ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124772025-04-30T06:46:32.309ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124782025-04-30T06:46:32.309ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124792025-04-30T06:46:32.309ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124802025-04-30T06:46:32.311ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124812025-04-30T06:46:32.312ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
124822025-04-30T06:46:32.314ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124832025-04-30T06:46:32.319ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124842025-04-30T06:46:32.319ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124852025-04-30T06:46:32.319ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124862025-04-30T06:46:32.319ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124872025-04-30T06:46:32.325ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
124882025-04-30T06:46:32.326ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
124892025-04-30T06:46:32.327ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ec bd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53835 req_id = c443f4fe-5896-4385-be2d-129efedf734c
124902025-04-30T06:46:32.327ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60605,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53835 req_id = c443f4fe-5896-4385-be2d-129efedf734c
124912025-04-30T06:46:32.327ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
124922025-04-30T06:46:32.328ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124932025-04-30T06:46:32.328ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53835 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c443f4fe-5896-4385-be2d-129efedf734c
124942025-04-30T06:46:32.328ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4457, tv_nsec: 127730439 } })
124952025-04-30T06:46:32.328ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
124962025-04-30T06:46:32.329ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 27 85 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e '...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61035 req_id = 34723f21-e1e2-419e-99e3-ddeb5ae37940
124972025-04-30T06:46:32.329ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10117,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61035 req_id = 34723f21-e1e2-419e-99e3-ddeb5ae37940
124982025-04-30T06:46:32.329ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124992025-04-30T06:46:32.329ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61035 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 34723f21-e1e2-419e-99e3-ddeb5ae37940
125002025-04-30T06:46:32.384ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125012025-04-30T06:46:32.384ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125022025-04-30T06:46:32.385ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125032025-04-30T06:46:32.385ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125042025-04-30T06:46:32.385ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125052025-04-30T06:46:32.386ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125062025-04-30T06:46:32.386ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 18 ac 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52215 req_id = 8a6c36a4-cd24-4a28-a5fd-df0aa3585456
125072025-04-30T06:46:32.386ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6316,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52215 req_id = 8a6c36a4-cd24-4a28-a5fd-df0aa3585456
125082025-04-30T06:46:32.386ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
125092025-04-30T06:46:32.387ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6316, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52215 req_id = 8a6c36a4-cd24-4a28-a5fd-df0aa3585456
125102025-04-30T06:46:32.387ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
125112025-04-30T06:46:32.387ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
125122025-04-30T06:46:32.387ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
125132025-04-30T06:46:32.389ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125142025-04-30T06:46:32.404ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125152025-04-30T06:46:32.405ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125162025-04-30T06:46:32.405ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125172025-04-30T06:46:32.405ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125182025-04-30T06:46:32.408ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad7e466d-480f-433a-a6a1-e5efd86f1e21", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125192025-04-30T06:46:32.408ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
125202025-04-30T06:46:32.409ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125212025-04-30T06:46:32.409ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125222025-04-30T06:46:32.410ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125232025-04-30T06:46:32.436ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4811667-905b-4b9c-9dc4-f0f83ed5a75e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125242025-04-30T06:46:32.436ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
125252025-04-30T06:46:32.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "deb41be1-0092-47d8-addf-257f1f735848", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125262025-04-30T06:46:32.450ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
125272025-04-30T06:46:32.453ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c20761fa-b251-47ec-8b7a-560cfc1e5f41", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125282025-04-30T06:46:32.453ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
125292025-04-30T06:46:32.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2945f78c-27e3-4e66-94e5-cc52a407c260", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125302025-04-30T06:46:32.456ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
125312025-04-30T06:46:32.458ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a1049f-d234-484c-a6d1-5966a7cab3ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125322025-04-30T06:46:32.458ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
125332025-04-30T06:46:32.460ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3ff6fe4-d21f-4a32-8d71-1fcf91179f33", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125342025-04-30T06:46:32.460ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
125352025-04-30T06:46:32.464ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
125362025-04-30T06:46:32.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99ccf358-1d62-4e7b-9a9c-38faa832a351", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125372025-04-30T06:46:32.466ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
125382025-04-30T06:46:32.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d108c6-69bc-4601-8e59-89eb6e90d1aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125392025-04-30T06:46:32.470ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
125402025-04-30T06:46:32.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a0c6af8-ff5b-4c73-a5ad-d8960756db57", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125412025-04-30T06:46:32.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
125422025-04-30T06:46:32.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bf2294b-7540-429e-9578-d9ee8838cbce", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125432025-04-30T06:46:32.474ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
125442025-04-30T06:46:32.476ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e38483d-4871-406b-bad1-b7b85310cb3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125452025-04-30T06:46:32.476ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
125462025-04-30T06:46:32.479ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94f2083f-1f1c-4f0b-859f-1198c129f3e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125472025-04-30T06:46:32.479ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
125482025-04-30T06:46:32.481ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e560c0d-9438-41ae-901c-99059f92020c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125492025-04-30T06:46:32.482ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
125502025-04-30T06:46:32.485ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce00daf-5621-4ca6-84a1-1c6639131be7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125512025-04-30T06:46:32.485ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
125522025-04-30T06:46:32.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de72aee2-61a7-4df6-aa45-62cff3ce51b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125532025-04-30T06:46:32.489ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
125542025-04-30T06:46:32.492ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d6a1a7-c6cc-4fde-a8b0-871ed5d1a06d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125552025-04-30T06:46:32.492ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
125562025-04-30T06:46:32.495ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be7ffb86-c403-4069-b635-41f23db324f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125572025-04-30T06:46:32.495ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
125582025-04-30T06:46:32.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d53282e-f735-4c50-8e46-4088da92b48e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125592025-04-30T06:46:32.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
125602025-04-30T06:46:32.499ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d9d0e7-3b29-49e8-b2aa-ab38e5098d13", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125612025-04-30T06:46:32.499ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
125622025-04-30T06:46:32.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04dc96b9-3472-427b-b5a5-e81aac339cf1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125632025-04-30T06:46:32.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
125642025-04-30T06:46:32.501ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffbd81c2-fe36-4407-b1c4-036b16edb7b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125652025-04-30T06:46:32.501ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
125662025-04-30T06:46:32.503ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a55bc272-cc4e-4547-a9f9-338dbb4b644d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125672025-04-30T06:46:32.503ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
125682025-04-30T06:46:32.504ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b13304aa-24e0-42df-8cfb-15c547c89925", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125692025-04-30T06:46:32.504ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
125702025-04-30T06:46:32.516ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125712025-04-30T06:46:32.516ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125722025-04-30T06:46:32.516ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125732025-04-30T06:46:32.517ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125742025-04-30T06:46:32.518ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125752025-04-30T06:46:32.518ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d34d751-8cd3-4b2c-9bf0-2c9852285d35", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125762025-04-30T06:46:32.519ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
125772025-04-30T06:46:32.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97bf1f6e-7d3c-40b7-acf7-8fb7551fa2c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125782025-04-30T06:46:32.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
125792025-04-30T06:46:32.522ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125802025-04-30T06:46:32.531ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125812025-04-30T06:46:32.531ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125822025-04-30T06:46:32.531ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125832025-04-30T06:46:32.531ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125842025-04-30T06:46:32.535ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125852025-04-30T06:46:32.535ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
125862025-04-30T06:46:32.537ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125872025-04-30T06:46:32.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87d3076e-6483-44d1-948a-d6bcab0fd04d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125882025-04-30T06:46:32.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
125892025-04-30T06:46:32.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b252ab4-187f-43f3-b40c-8506d5ebb991", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125902025-04-30T06:46:32.541ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
125912025-04-30T06:46:32.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54335fba-b52a-4a38-bf04-cb5af343dd61", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125922025-04-30T06:46:32.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
125932025-04-30T06:46:32.548ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb9d93f-7b79-426a-8ec7-5834ef9d4629", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125942025-04-30T06:46:32.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
125952025-04-30T06:46:32.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c0b804-8f15-4830-8ac7-841064af26d1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125962025-04-30T06:46:32.551ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
125972025-04-30T06:46:32.554ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43fb0a14-b505-4167-a566-17d5503fb131", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
125982025-04-30T06:46:32.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
125992025-04-30T06:46:32.556ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba9c7ec0-c6d1-4661-9f60-199d06733280", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126002025-04-30T06:46:32.556ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
126012025-04-30T06:46:32.559ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e20ea520-a44f-4e38-9be0-a625520d494f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126022025-04-30T06:46:32.559ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
126032025-04-30T06:46:32.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7efa87b7-db6d-4f22-9b79-e1ffdde7d649", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126042025-04-30T06:46:32.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
126052025-04-30T06:46:32.565ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "010c73a5-f27f-47cc-8309-08b8ff211ea9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126062025-04-30T06:46:32.566ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
126072025-04-30T06:46:32.570ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
126082025-04-30T06:46:32.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea61d919-5e1d-4542-a61d-d084324872a2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126092025-04-30T06:46:32.573ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
126102025-04-30T06:46:32.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b98ef02-37bf-4bd6-a219-586dc1a2c0af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126112025-04-30T06:46:32.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
126122025-04-30T06:46:32.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d6c9455-8500-48ab-918a-936cec71092a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126132025-04-30T06:46:32.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
126142025-04-30T06:46:32.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6635d251-b4d7-4970-bad7-2ebbc2a11e3f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126152025-04-30T06:46:32.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
126162025-04-30T06:46:32.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de092781-a3f6-4e7f-bc1c-b83dd9715db8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126172025-04-30T06:46:32.582ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
126182025-04-30T06:46:32.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb8bf0c-36d9-48fc-bc84-1568d71ce433", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126192025-04-30T06:46:32.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
126202025-04-30T06:46:32.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc9fba22-387b-4559-859c-babeb0b96dfb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126212025-04-30T06:46:32.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
126222025-04-30T06:46:32.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3854dab-7073-4dec-b79f-0d15419ada00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126232025-04-30T06:46:32.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
126242025-04-30T06:46:32.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cf018c8-32eb-4a0e-ab75-37709e7b2be3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:31 GMT"} })
126252025-04-30T06:46:32.587ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126262025-04-30T06:46:32.587ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126272025-04-30T06:46:32.587ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126282025-04-30T06:46:32.587ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126292025-04-30T06:46:32.588ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126302025-04-30T06:46:32.591ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126312025-04-30T06:46:32.600ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126322025-04-30T06:46:32.600ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126332025-04-30T06:46:32.600ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126342025-04-30T06:46:32.600ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126352025-04-30T06:46:32.600ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126362025-04-30T06:46:32.601ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126372025-04-30T06:46:32.604ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126382025-04-30T06:46:32.612ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126392025-04-30T06:46:32.612ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126402025-04-30T06:46:32.612ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126412025-04-30T06:46:32.613ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126422025-04-30T06:46:32.623ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126432025-04-30T06:46:32.623ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126442025-04-30T06:46:32.623ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126452025-04-30T06:46:32.623ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
126462025-04-30T06:46:32.624ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126472025-04-30T06:46:32.691ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
126482025-04-30T06:46:32.691ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
126492025-04-30T06:46:32.692ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:62703 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126502025-04-30T06:46:32.692ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126512025-04-30T06:46:32.692ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126522025-04-30T06:46:32.693ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:62703 req_id = 87be2b60-79ed-4514-b9cb-c66ebc391f0e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126532025-04-30T06:46:32.695ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 2627 local_addr = [::1]:54590 method = GET remote_addr = [::1]:62703 req_id = 87be2b60-79ed-4514-b9cb-c66ebc391f0e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126542025-04-30T06:46:32.704ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126552025-04-30T06:46:32.720ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126562025-04-30T06:46:32.720ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126572025-04-30T06:46:32.720ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
126582025-04-30T06:46:32.721ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126592025-04-30T06:46:32.721ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126602025-04-30T06:46:32.728ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126612025-04-30T06:46:32.728ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126622025-04-30T06:46:32.728ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126632025-04-30T06:46:32.728ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "87be2b60-79ed-4514-b9cb-c66ebc391f0e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
126642025-04-30T06:46:32.730ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126652025-04-30T06:46:32.730ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126662025-04-30T06:46:32.730ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
126672025-04-30T06:46:32.732ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126682025-04-30T06:46:32.732ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126692025-04-30T06:46:32.732ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126702025-04-30T06:46:32.732ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126712025-04-30T06:46:32.733ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126722025-04-30T06:46:32.736ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126732025-04-30T06:46:32.736ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
126742025-04-30T06:46:32.739ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126752025-04-30T06:46:32.739ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126762025-04-30T06:46:32.745ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126772025-04-30T06:46:32.745ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126782025-04-30T06:46:32.745ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126792025-04-30T06:46:32.746ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126802025-04-30T06:46:32.779ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
126812025-04-30T06:46:32.779ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
126822025-04-30T06:46:32.779ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 470.77556ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
126832025-04-30T06:46:32.779ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
126842025-04-30T06:46:32.780ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126852025-04-30T06:46:32.780ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126862025-04-30T06:46:32.780ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126872025-04-30T06:46:32.780ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126882025-04-30T06:46:32.781ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126892025-04-30T06:46:32.786ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126902025-04-30T06:46:32.799ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126912025-04-30T06:46:32.799ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126922025-04-30T06:46:32.799ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126932025-04-30T06:46:32.802ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126942025-04-30T06:46:32.803ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
126952025-04-30T06:46:32.805ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
126962025-04-30T06:46:32.806ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
126972025-04-30T06:46:32.807ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8f ed 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59415 req_id = 6dccb688-573c-42b0-8ed7-1fda8141845c
126982025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36845,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59415 req_id = 6dccb688-573c-42b0-8ed7-1fda8141845c
126992025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
127002025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127012025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59415 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6dccb688-573c-42b0-8ed7-1fda8141845c
127022025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4457, tv_nsec: 606413093 } })
127032025-04-30T06:46:32.807ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
127042025-04-30T06:46:32.807ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b8 b0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53025 req_id = 1845127f-a218-44ad-9860-8961f40bd8ce
127052025-04-30T06:46:32.807ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47280,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53025 req_id = 1845127f-a218-44ad-9860-8961f40bd8ce
127062025-04-30T06:46:32.808ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127072025-04-30T06:46:32.808ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53025 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1845127f-a218-44ad-9860-8961f40bd8ce
127082025-04-30T06:46:32.849ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127092025-04-30T06:46:32.850ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 67 7c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c g|..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59532 req_id = 8da4a902-c273-4c1b-832e-d5dacac73305
127102025-04-30T06:46:32.850ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26492,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59532 req_id = 8da4a902-c273-4c1b-832e-d5dacac73305
127112025-04-30T06:46:32.850ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
127122025-04-30T06:46:32.850ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26492, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59532 req_id = 8da4a902-c273-4c1b-832e-d5dacac73305
127132025-04-30T06:46:32.850ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
127142025-04-30T06:46:32.850ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
127152025-04-30T06:46:32.850ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
127162025-04-30T06:46:32.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8601953-01ea-4318-b68f-ab237fe00e0c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127172025-04-30T06:46:32.852ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
127182025-04-30T06:46:32.854ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa99ea40-bcd1-464a-9b60-644c7a75fd88", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127192025-04-30T06:46:32.854ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
127202025-04-30T06:46:32.854ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127212025-04-30T06:46:32.854ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127222025-04-30T06:46:32.854ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127232025-04-30T06:46:32.854ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
127242025-04-30T06:46:32.855ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127252025-04-30T06:46:32.857ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127262025-04-30T06:46:32.863ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127272025-04-30T06:46:32.863ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127282025-04-30T06:46:32.863ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
127292025-04-30T06:46:32.864ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127302025-04-30T06:46:32.865ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e7418ef-ab6e-4fbd-afc9-9ed5e210c2d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127312025-04-30T06:46:32.865ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
127322025-04-30T06:46:32.866ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127332025-04-30T06:46:32.866ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
127342025-04-30T06:46:32.867ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127352025-04-30T06:46:32.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b90d18e-3dbb-48df-99fd-dee67ae6c27a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127362025-04-30T06:46:32.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
127372025-04-30T06:46:32.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80839025-8c68-47e3-8d1b-861a0d316e29", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127382025-04-30T06:46:32.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
127392025-04-30T06:46:32.876ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b9cd872-1264-446e-b5b7-96eeb64d8f30", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127402025-04-30T06:46:32.877ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
127412025-04-30T06:46:32.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04911e28-f61c-4fb7-aca1-e4f5622d43f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127422025-04-30T06:46:32.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
127432025-04-30T06:46:32.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a8298f1-c339-4795-95a3-bcc67a10fb1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127442025-04-30T06:46:32.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
127452025-04-30T06:46:32.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d20a990-63be-44aa-8df0-f0ce67d0b72f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127462025-04-30T06:46:32.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
127472025-04-30T06:46:32.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17c2b76-a1a5-4471-9391-fa77893f9e67", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127482025-04-30T06:46:32.888ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
127492025-04-30T06:46:32.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31f6f053-fcc5-4a3d-94ba-ec419d0fd6f8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127502025-04-30T06:46:32.890ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
127512025-04-30T06:46:32.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee3b39a6-43d2-4722-944e-7806c45fecb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127522025-04-30T06:46:32.892ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
127532025-04-30T06:46:32.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48fc9ab-958a-4cd2-be63-6827baa4891a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127542025-04-30T06:46:32.895ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
127552025-04-30T06:46:32.896ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
127562025-04-30T06:46:32.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2d22ca-ae65-411b-8266-867ffe39e819", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127572025-04-30T06:46:32.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
127582025-04-30T06:46:32.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f41927a-a504-4893-af16-4ed779954d67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127592025-04-30T06:46:32.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
127602025-04-30T06:46:32.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c45f53-1b1f-4349-b70c-c106a923fd80", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127612025-04-30T06:46:32.906ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
127622025-04-30T06:46:32.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c38da60-18d1-4464-9c69-b05be9a65aa1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127632025-04-30T06:46:32.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
127642025-04-30T06:46:32.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "137a86dc-caad-4eb3-8b90-89e357f86236", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127652025-04-30T06:46:32.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
127662025-04-30T06:46:32.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e2ddc5a-9b18-42e4-8447-c135bedcf650", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127672025-04-30T06:46:32.915ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
127682025-04-30T06:46:32.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b86316-d9bd-4e7f-8fdf-8f3f7eb29263", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127692025-04-30T06:46:32.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
127702025-04-30T06:46:32.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09518706-9680-4e52-a949-651e42b4cf5d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127712025-04-30T06:46:32.917ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
127722025-04-30T06:46:32.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "faee8a27-c2a4-467c-86e9-dce1d213f9f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127732025-04-30T06:46:32.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
127742025-04-30T06:46:32.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2857bcb7-7b8d-46ff-abca-4c587ca07a6f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127752025-04-30T06:46:32.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
127762025-04-30T06:46:32.922ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89d590f1-587d-4836-8425-7a2d70993cd8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127772025-04-30T06:46:32.922ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
127782025-04-30T06:46:32.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b63974-81d2-4a57-8202-13d7163f0b1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127792025-04-30T06:46:32.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
127802025-04-30T06:46:32.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca8b37e1-b35e-416f-98c6-172568d372e1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127812025-04-30T06:46:32.928ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
127822025-04-30T06:46:32.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97fc7d0-dcba-4503-91b2-7ada3d4d4535", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127832025-04-30T06:46:32.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
127842025-04-30T06:46:32.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93a7a875-0f4b-41f8-92ac-b6be4921a749", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127852025-04-30T06:46:32.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
127862025-04-30T06:46:32.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "437f9af8-be9f-4d28-9607-a01766a094d3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127872025-04-30T06:46:32.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
127882025-04-30T06:46:32.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32b5bff6-be49-41a2-b039-3924b29b858b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127892025-04-30T06:46:32.938ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
127902025-04-30T06:46:32.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9ed9c3-35db-4289-8415-1ea56ad2e7d3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127912025-04-30T06:46:32.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
127922025-04-30T06:46:32.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31705f05-9d90-4bef-b73f-e4b43f5c0a4b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127932025-04-30T06:46:32.941ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
127942025-04-30T06:46:32.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3e9d44f-f343-4164-ab66-b4ed7c6c6d07", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127952025-04-30T06:46:32.942ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
127962025-04-30T06:46:32.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "768a0a84-87fb-4edb-b87b-eeadaea8cbf6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127972025-04-30T06:46:32.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
127982025-04-30T06:46:32.945ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8ad4047-4a9f-4f91-996b-d2ff631962a6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
127992025-04-30T06:46:32.945ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
128002025-04-30T06:46:32.947ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128012025-04-30T06:46:32.947ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128022025-04-30T06:46:32.947ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128032025-04-30T06:46:32.947ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128042025-04-30T06:46:32.948ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128052025-04-30T06:46:32.949ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9583bcae-181f-499c-978d-131c6a0ef0e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128062025-04-30T06:46:32.950ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
128072025-04-30T06:46:32.951ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128082025-04-30T06:46:32.957ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128092025-04-30T06:46:32.957ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128102025-04-30T06:46:32.957ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128112025-04-30T06:46:32.958ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128122025-04-30T06:46:32.960ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a598011b-fda7-4200-95ec-276f32d21cb6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128132025-04-30T06:46:32.960ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
128142025-04-30T06:46:32.960ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128152025-04-30T06:46:32.960ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128162025-04-30T06:46:32.961ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128172025-04-30T06:46:32.964ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b4bc3e-7709-405c-8cf3-cbea747eebb7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128182025-04-30T06:46:32.964ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
128192025-04-30T06:46:32.968ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef4b810-dd95-4173-9065-3f01c4164924", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128202025-04-30T06:46:32.968ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
128212025-04-30T06:46:32.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d84580e-2f11-438b-89fd-57f23ccafc49", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128222025-04-30T06:46:32.972ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
128232025-04-30T06:46:32.974ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d08e760d-aa45-456b-ab4f-65866a211cbb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128242025-04-30T06:46:32.974ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
128252025-04-30T06:46:32.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35506ef6-f506-4f56-9e1e-223de63d80f8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128262025-04-30T06:46:32.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
128272025-04-30T06:46:32.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe3f2e9-f512-4a23-8653-e837a43b34f3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128282025-04-30T06:46:32.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
128292025-04-30T06:46:32.979ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7fe643e-2211-4e5f-9c15-e10d8cd83834", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128302025-04-30T06:46:32.979ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
128312025-04-30T06:46:32.980ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128322025-04-30T06:46:32.980ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128332025-04-30T06:46:32.980ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
128342025-04-30T06:46:32.980ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
128352025-04-30T06:46:33.024ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9251f842-9ea7-48c2-a752-99065581f009", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128362025-04-30T06:46:33.024ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128372025-04-30T06:46:33.024ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128382025-04-30T06:46:33.024ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128392025-04-30T06:46:33.024ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128402025-04-30T06:46:33.025ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128412025-04-30T06:46:33.030ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128422025-04-30T06:46:33.040ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128432025-04-30T06:46:33.040ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128442025-04-30T06:46:33.040ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128452025-04-30T06:46:33.040ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128462025-04-30T06:46:33.040ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128472025-04-30T06:46:33.040ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128482025-04-30T06:46:33.040ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
128492025-04-30T06:46:33.040ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
128502025-04-30T06:46:33.045ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
128512025-04-30T06:46:33.046ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128522025-04-30T06:46:33.052ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128532025-04-30T06:46:33.052ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128542025-04-30T06:46:33.052ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128552025-04-30T06:46:33.054ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128562025-04-30T06:46:33.055ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
128572025-04-30T06:46:33.055ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
128582025-04-30T06:46:33.105ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
128592025-04-30T06:46:33.105ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
128602025-04-30T06:46:33.106ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128612025-04-30T06:46:33.106ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128622025-04-30T06:46:33.106ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128632025-04-30T06:46:33.106ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128642025-04-30T06:46:33.107ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128652025-04-30T06:46:33.107ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:47370 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128662025-04-30T06:46:33.107ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
128672025-04-30T06:46:33.108ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:47370 req_id = c38b1574-1f74-4e7f-bcad-41e45022c204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128682025-04-30T06:46:33.108ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 525 local_addr = [::1]:54590 method = GET remote_addr = [::1]:47370 req_id = c38b1574-1f74-4e7f-bcad-41e45022c204 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128692025-04-30T06:46:33.108ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c38b1574-1f74-4e7f-bcad-41e45022c204", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
128702025-04-30T06:46:33.110ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128712025-04-30T06:46:33.110ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128722025-04-30T06:46:33.110ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
128732025-04-30T06:46:33.114ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128742025-04-30T06:46:33.114ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128752025-04-30T06:46:33.115ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128762025-04-30T06:46:33.115ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128772025-04-30T06:46:33.116ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128782025-04-30T06:46:33.116ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128792025-04-30T06:46:33.125ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128802025-04-30T06:46:33.126ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128812025-04-30T06:46:33.126ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128822025-04-30T06:46:33.126ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128832025-04-30T06:46:33.128ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128842025-04-30T06:46:33.129ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
128852025-04-30T06:46:33.129ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128862025-04-30T06:46:33.130ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128872025-04-30T06:46:33.136ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128882025-04-30T06:46:33.136ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128892025-04-30T06:46:33.136ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128902025-04-30T06:46:33.138ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128912025-04-30T06:46:33.189ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128922025-04-30T06:46:33.190ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128932025-04-30T06:46:33.190ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 410.766035ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128942025-04-30T06:46:33.190ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
128952025-04-30T06:46:33.190ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128962025-04-30T06:46:33.190ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128972025-04-30T06:46:33.190ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128982025-04-30T06:46:33.190ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128992025-04-30T06:46:33.192ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129002025-04-30T06:46:33.196ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
129012025-04-30T06:46:33.197ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129022025-04-30T06:46:33.203ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129032025-04-30T06:46:33.204ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129042025-04-30T06:46:33.204ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129052025-04-30T06:46:33.204ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129062025-04-30T06:46:33.208ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
129072025-04-30T06:46:33.208ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129082025-04-30T06:46:33.209ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 04 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51013 req_id = 91ad02ae-e540-47d7-8481-37a5489bd80b
129092025-04-30T06:46:33.209ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45572,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51013 req_id = 91ad02ae-e540-47d7-8481-37a5489bd80b
129102025-04-30T06:46:33.209ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
129112025-04-30T06:46:33.210ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129122025-04-30T06:46:33.210ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51013 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 91ad02ae-e540-47d7-8481-37a5489bd80b
129132025-04-30T06:46:33.210ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4458, tv_nsec: 9671549 } })
129142025-04-30T06:46:33.210ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
129152025-04-30T06:46:33.211ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd 6f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .o..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52872 req_id = 3477a5b7-46b1-47c5-b97d-157fe09db0e8
129162025-04-30T06:46:33.211ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48495,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52872 req_id = 3477a5b7-46b1-47c5-b97d-157fe09db0e8
129172025-04-30T06:46:33.211ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129182025-04-30T06:46:33.211ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52872 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3477a5b7-46b1-47c5-b97d-157fe09db0e8
129192025-04-30T06:46:33.285ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
129202025-04-30T06:46:33.286ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129212025-04-30T06:46:33.286ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129222025-04-30T06:46:33.286ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129232025-04-30T06:46:33.286ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
129242025-04-30T06:46:33.287ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129252025-04-30T06:46:33.288ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b4 2b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .+..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55231 req_id = 1f021a6d-16b9-498c-bc04-f1aab2ee85a1
129262025-04-30T06:46:33.288ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46123,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55231 req_id = 1f021a6d-16b9-498c-bc04-f1aab2ee85a1
129272025-04-30T06:46:33.288ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
129282025-04-30T06:46:33.288ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46123, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55231 req_id = 1f021a6d-16b9-498c-bc04-f1aab2ee85a1
129292025-04-30T06:46:33.288ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
129302025-04-30T06:46:33.288ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
129312025-04-30T06:46:33.289ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
129322025-04-30T06:46:33.290ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129332025-04-30T06:46:33.298ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129342025-04-30T06:46:33.298ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129352025-04-30T06:46:33.298ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
129362025-04-30T06:46:33.299ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129372025-04-30T06:46:33.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a51ad22-bedb-4fa6-aeba-78b64f183ba7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129382025-04-30T06:46:33.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
129392025-04-30T06:46:33.302ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129402025-04-30T06:46:33.302ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
129412025-04-30T06:46:33.304ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129422025-04-30T06:46:33.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ec8dc5f-b5e6-476d-84ff-cc1f28458efb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129432025-04-30T06:46:33.305ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
129442025-04-30T06:46:33.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf503d4-53f1-47a6-912c-8a59169e4a00", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129452025-04-30T06:46:33.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
129462025-04-30T06:46:33.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31b96367-514a-4f97-8d2c-b0193eef4f7b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129472025-04-30T06:46:33.309ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
129482025-04-30T06:46:33.312ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32983746-12f6-45db-962d-3638a3a82d76", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129492025-04-30T06:46:33.314ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
129502025-04-30T06:46:33.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b70db8d-8362-4dfc-bf71-b16dc692ca75", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129512025-04-30T06:46:33.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
129522025-04-30T06:46:33.320ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7586f4ce-d6f7-45ba-a192-ccd3456c6da9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129532025-04-30T06:46:33.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
129542025-04-30T06:46:33.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab3385a-131d-4605-9942-ab4d239da37c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129552025-04-30T06:46:33.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
129562025-04-30T06:46:33.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a2ea067-a3ba-4d16-84ba-32a173e89c20", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129572025-04-30T06:46:33.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
129582025-04-30T06:46:33.335ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf95279e-29a5-4c60-a22e-0b5b4b3602fd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129592025-04-30T06:46:33.335ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
129602025-04-30T06:46:33.338ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae08190b-fd49-4cf5-a51c-5fd3f4710063", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129612025-04-30T06:46:33.339ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
129622025-04-30T06:46:33.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e09f36-5484-4f47-8851-752ec551beb4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129632025-04-30T06:46:33.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
129642025-04-30T06:46:33.350ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
129652025-04-30T06:46:33.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16ee8720-2213-4f99-ac99-e7281f24a92e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129662025-04-30T06:46:33.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
129672025-04-30T06:46:33.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "372dae5b-5151-4097-87e8-3f7e71fdb08a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129682025-04-30T06:46:33.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
129692025-04-30T06:46:33.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22dc0d71-fddb-4fe6-8020-9c93e82662a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129702025-04-30T06:46:33.357ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
129712025-04-30T06:46:33.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e333918-f543-4cff-865d-f48b98e3d568", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129722025-04-30T06:46:33.367ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
129732025-04-30T06:46:33.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba6e901-d686-4dee-b96d-3361e0885811", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129742025-04-30T06:46:33.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
129752025-04-30T06:46:33.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f10d033-aa3e-406f-ae25-76b7ddb33d03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129762025-04-30T06:46:33.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
129772025-04-30T06:46:33.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6383959-fee3-4660-a8df-5ada67f2eb96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129782025-04-30T06:46:33.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
129792025-04-30T06:46:33.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a79b7f6-0f8e-484c-bc99-43fea472de51", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129802025-04-30T06:46:33.379ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
129812025-04-30T06:46:33.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b946d8-f1e9-4e95-ab1d-3638305d703c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129822025-04-30T06:46:33.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
129832025-04-30T06:46:33.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f6ef0d3-a11c-442b-8aff-d08a4a0d6d44", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129842025-04-30T06:46:33.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
129852025-04-30T06:46:33.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65327892-413e-4fdb-a7fb-03e4ad361218", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129862025-04-30T06:46:33.383ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
129872025-04-30T06:46:33.385ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c3ecca1-d407-4420-8c10-284a5ce21a80", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129882025-04-30T06:46:33.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
129892025-04-30T06:46:33.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0095c52a-9a99-488d-a620-bb1465e722de", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129902025-04-30T06:46:33.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
129912025-04-30T06:46:33.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70c329ae-8db6-4372-afb2-d2e0d8774906", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129922025-04-30T06:46:33.390ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
129932025-04-30T06:46:33.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "892e6b35-7a66-4787-9a1e-fdc84b41c0f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129942025-04-30T06:46:33.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
129952025-04-30T06:46:33.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1947b2a8-ec0c-475f-818b-6078d2215a5f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129962025-04-30T06:46:33.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
129972025-04-30T06:46:33.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ed181d0-e1a2-40a0-8550-eecd94069c2e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
129982025-04-30T06:46:33.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
129992025-04-30T06:46:33.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9f8df46-a266-49b3-b7f3-ff4e9b3a0333", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130002025-04-30T06:46:33.400ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
130012025-04-30T06:46:33.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc487832-295e-4a8a-90a4-6b47ff0be44c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130022025-04-30T06:46:33.401ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
130032025-04-30T06:46:33.401ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130042025-04-30T06:46:33.402ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130052025-04-30T06:46:33.402ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130062025-04-30T06:46:33.402ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
130072025-04-30T06:46:33.403ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130082025-04-30T06:46:33.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ac3799-ca7b-4409-8292-ceb142f7aa1a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130092025-04-30T06:46:33.404ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
130102025-04-30T06:46:33.405ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f4fabe-9be2-4d75-a30e-72aaa07925f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130112025-04-30T06:46:33.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
130122025-04-30T06:46:33.406ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130132025-04-30T06:46:33.418ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130142025-04-30T06:46:33.418ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130152025-04-30T06:46:33.418ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
130162025-04-30T06:46:33.420ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130172025-04-30T06:46:33.434ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130182025-04-30T06:46:33.434ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
130192025-04-30T06:46:33.435ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130202025-04-30T06:46:33.436ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36f9cab6-562b-4d09-9ca0-f2a973683b9e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130212025-04-30T06:46:33.436ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
130222025-04-30T06:46:33.471ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe0848d5-8d44-417f-9889-72194e789e7e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130232025-04-30T06:46:33.472ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
130242025-04-30T06:46:33.497ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c59b83f6-1805-4458-993a-c579c95956d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130252025-04-30T06:46:33.498ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
130262025-04-30T06:46:33.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3405d54b-98d0-40bb-aeab-b23bff8b3a0c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130272025-04-30T06:46:33.500ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
130282025-04-30T06:46:33.502ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d41cdf6-5d79-455c-825f-297dd9b899aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130292025-04-30T06:46:33.503ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
130302025-04-30T06:46:33.555ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
130312025-04-30T06:46:33.556ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130322025-04-30T06:46:33.557ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
130332025-04-30T06:46:33.557ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 7a4f61cd-faac-428d-ad43-b1de9a287ef5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130342025-04-30T06:46:33.557ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 7a4f61cd-faac-428d-ad43-b1de9a287ef5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130352025-04-30T06:46:33.557ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 7a4f61cd-faac-428d-ad43-b1de9a287ef5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130362025-04-30T06:46:33.559ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 7a4f61cd-faac-428d-ad43-b1de9a287ef5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130372025-04-30T06:46:33.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "005c58f3-fb4e-4a89-b7b9-a584c6c70285", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130382025-04-30T06:46:33.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
130392025-04-30T06:46:33.564ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 7485 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 7a4f61cd-faac-428d-ad43-b1de9a287ef5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130402025-04-30T06:46:33.565ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a4f61cd-faac-428d-ad43-b1de9a287ef5", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130412025-04-30T06:46:33.565ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130422025-04-30T06:46:33.566ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f7f026e9-f8f6-4f77-9d54-734ba0af208c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130432025-04-30T06:46:33.566ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f7f026e9-f8f6-4f77-9d54-734ba0af208c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130442025-04-30T06:46:33.567ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f7f026e9-f8f6-4f77-9d54-734ba0af208c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130452025-04-30T06:46:33.569ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f7f026e9-f8f6-4f77-9d54-734ba0af208c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130462025-04-30T06:46:33.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e38b8c91-61c0-4b5e-933e-4b554a234e3f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130472025-04-30T06:46:33.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
130482025-04-30T06:46:33.571ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 5070 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f7f026e9-f8f6-4f77-9d54-734ba0af208c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
130492025-04-30T06:46:33.572ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7f026e9-f8f6-4f77-9d54-734ba0af208c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130502025-04-30T06:46:33.572ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
130512025-04-30T06:46:33.572ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130522025-04-30T06:46:33.572ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
130532025-04-30T06:46:33.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf1897e2-d5dd-452f-ac98-2a941badc60c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130542025-04-30T06:46:33.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
130552025-04-30T06:46:33.578ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7795e40c-4dbf-4c13-917e-c6f109e38262", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130562025-04-30T06:46:33.579ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
130572025-04-30T06:46:33.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0429b9d-f629-4f72-9a58-7aacdc7b0c3c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130582025-04-30T06:46:33.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
130592025-04-30T06:46:33.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32798b14-fa94-4360-823b-a98210103c2a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130602025-04-30T06:46:33.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
130612025-04-30T06:46:33.586ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace08cc0-5b58-4376-ae83-3e06c3a635f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:32 GMT"} })
130622025-04-30T06:46:33.586ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130632025-04-30T06:46:33.586ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130642025-04-30T06:46:33.586ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130652025-04-30T06:46:33.586ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130662025-04-30T06:46:33.587ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130672025-04-30T06:46:33.592ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130682025-04-30T06:46:33.600ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130692025-04-30T06:46:33.600ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130702025-04-30T06:46:33.600ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130712025-04-30T06:46:33.601ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130722025-04-30T06:46:33.601ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130732025-04-30T06:46:33.602ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130742025-04-30T06:46:33.606ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130752025-04-30T06:46:33.614ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130762025-04-30T06:46:33.614ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130772025-04-30T06:46:33.614ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130782025-04-30T06:46:33.619ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130792025-04-30T06:46:33.619ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130802025-04-30T06:46:33.619ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130812025-04-30T06:46:33.620ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130822025-04-30T06:46:33.620ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
130832025-04-30T06:46:33.620ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130842025-04-30T06:46:33.708ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
130852025-04-30T06:46:33.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
130862025-04-30T06:46:33.710ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:34103 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130872025-04-30T06:46:33.711ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130882025-04-30T06:46:33.711ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130892025-04-30T06:46:33.712ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130902025-04-30T06:46:33.719ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130912025-04-30T06:46:33.719ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130922025-04-30T06:46:33.719ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
130932025-04-30T06:46:33.720ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130942025-04-30T06:46:33.720ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:34103 req_id = 320e0905-ca01-4bd5-8658-fe7d102cf98a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130952025-04-30T06:46:33.725ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 5545 local_addr = [::1]:54590 method = GET remote_addr = [::1]:34103 req_id = 320e0905-ca01-4bd5-8658-fe7d102cf98a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130962025-04-30T06:46:33.726ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130972025-04-30T06:46:33.727ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130982025-04-30T06:46:33.727ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
130992025-04-30T06:46:33.727ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131002025-04-30T06:46:33.727ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "320e0905-ca01-4bd5-8658-fe7d102cf98a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131012025-04-30T06:46:33.731ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131022025-04-30T06:46:33.731ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131032025-04-30T06:46:33.731ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
131042025-04-30T06:46:33.735ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131052025-04-30T06:46:33.735ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131062025-04-30T06:46:33.735ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131072025-04-30T06:46:33.735ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131082025-04-30T06:46:33.736ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131092025-04-30T06:46:33.740ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131102025-04-30T06:46:33.740ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
131112025-04-30T06:46:33.740ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131122025-04-30T06:46:33.740ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131132025-04-30T06:46:33.744ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131142025-04-30T06:46:33.745ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131152025-04-30T06:46:33.746ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131162025-04-30T06:46:33.747ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131172025-04-30T06:46:33.778ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
131182025-04-30T06:46:33.779ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
131192025-04-30T06:46:33.779ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 588.555639ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
131202025-04-30T06:46:33.779ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
131212025-04-30T06:46:33.779ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131222025-04-30T06:46:33.779ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131232025-04-30T06:46:33.779ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131242025-04-30T06:46:33.779ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131252025-04-30T06:46:33.780ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131262025-04-30T06:46:33.783ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
131272025-04-30T06:46:33.785ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131282025-04-30T06:46:33.788ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131292025-04-30T06:46:33.789ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131302025-04-30T06:46:33.789ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131312025-04-30T06:46:33.789ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131322025-04-30T06:46:33.792ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
131332025-04-30T06:46:33.792ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
131342025-04-30T06:46:33.793ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 ce 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51185 req_id = 1746f735-e724-404c-bf5c-dcc14868b9d2
131352025-04-30T06:46:33.793ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13518,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51185 req_id = 1746f735-e724-404c-bf5c-dcc14868b9d2
131362025-04-30T06:46:33.793ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
131372025-04-30T06:46:33.793ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131382025-04-30T06:46:33.793ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51185 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1746f735-e724-404c-bf5c-dcc14868b9d2
131392025-04-30T06:46:33.794ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4458, tv_nsec: 593080593 } })
131402025-04-30T06:46:33.794ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
131412025-04-30T06:46:33.794ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: db 7d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .}..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50859 req_id = 356e6306-e7eb-4987-a86d-73a5938ae40d
131422025-04-30T06:46:33.794ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56189,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50859 req_id = 356e6306-e7eb-4987-a86d-73a5938ae40d
131432025-04-30T06:46:33.794ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131442025-04-30T06:46:33.794ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50859 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 356e6306-e7eb-4987-a86d-73a5938ae40d
131452025-04-30T06:46:33.849ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
131462025-04-30T06:46:33.849ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131472025-04-30T06:46:33.849ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131482025-04-30T06:46:33.849ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131492025-04-30T06:46:33.849ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
131502025-04-30T06:46:33.850ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131512025-04-30T06:46:33.850ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b5 c2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57904 req_id = d82bd150-5c05-4f40-85f7-db0508fbe61f
131522025-04-30T06:46:33.850ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46530,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57904 req_id = d82bd150-5c05-4f40-85f7-db0508fbe61f
131532025-04-30T06:46:33.851ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
131542025-04-30T06:46:33.851ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46530, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57904 req_id = d82bd150-5c05-4f40-85f7-db0508fbe61f
131552025-04-30T06:46:33.851ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
131562025-04-30T06:46:33.851ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
131572025-04-30T06:46:33.851ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
131582025-04-30T06:46:33.856ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131592025-04-30T06:46:33.862ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131602025-04-30T06:46:33.862ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131612025-04-30T06:46:33.862ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
131622025-04-30T06:46:33.862ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131632025-04-30T06:46:33.862ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36f6ec6-941b-4370-bbb1-cb684fd2e6b7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131642025-04-30T06:46:33.863ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
131652025-04-30T06:46:33.865ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131662025-04-30T06:46:33.865ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
131672025-04-30T06:46:33.865ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131682025-04-30T06:46:33.866ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b661c5e-0913-4ff0-b885-89c8782687fd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131692025-04-30T06:46:33.866ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
131702025-04-30T06:46:33.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6e14be-9bce-44d6-9983-5e66bdddf59d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131712025-04-30T06:46:33.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
131722025-04-30T06:46:33.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad280111-df8e-4bc4-82e0-998fddd10215", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131732025-04-30T06:46:33.875ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
131742025-04-30T06:46:33.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f5ef8c-b1f5-47cd-a6ab-bd67a3fdb317", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131752025-04-30T06:46:33.879ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
131762025-04-30T06:46:33.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a180f26e-7bfd-4443-a035-c0c1756a966b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131772025-04-30T06:46:33.884ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
131782025-04-30T06:46:33.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f214812-6362-4843-a70f-34accd117161", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131792025-04-30T06:46:33.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
131802025-04-30T06:46:33.897ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
131812025-04-30T06:46:33.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "657a04a8-b40d-4c05-bf4b-25ae16369b35", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131822025-04-30T06:46:33.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
131832025-04-30T06:46:33.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdb5510e-87b6-4e92-be79-723eb7140584", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131842025-04-30T06:46:33.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
131852025-04-30T06:46:33.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41758596-9280-4458-9fc8-00e9b3c9ffa9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131862025-04-30T06:46:33.902ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
131872025-04-30T06:46:33.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c2d9b90-8c89-453b-8abb-f2432721a041", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131882025-04-30T06:46:33.903ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
131892025-04-30T06:46:33.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "876c3245-db79-4316-b852-6fc2a11699ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131902025-04-30T06:46:33.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
131912025-04-30T06:46:33.907ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "623ef151-44b5-4a10-acdc-319f9bb2b3ab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131922025-04-30T06:46:33.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
131932025-04-30T06:46:33.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2b61258-5489-4bc5-868d-4ed524ed48db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131942025-04-30T06:46:33.912ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
131952025-04-30T06:46:33.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb290ad8-b54f-43d7-ae29-9e43bc4f8978", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131962025-04-30T06:46:33.916ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
131972025-04-30T06:46:33.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9395dc38-4df8-4591-99bc-5f027a257fef", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
131982025-04-30T06:46:33.920ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
131992025-04-30T06:46:33.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebc92fca-7f3e-4870-8b36-9c10420a97a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132002025-04-30T06:46:33.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
132012025-04-30T06:46:33.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13143f4b-88b0-4ea8-9786-559f5564376f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132022025-04-30T06:46:33.927ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
132032025-04-30T06:46:33.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78a2b058-cf05-404b-b23b-2f3fac66b778", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132042025-04-30T06:46:33.931ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
132052025-04-30T06:46:33.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b14094cb-aaf5-483b-a487-2bf06efce205", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132062025-04-30T06:46:33.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
132072025-04-30T06:46:33.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "434f851b-d59f-494f-a54c-1cc08ed8198e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132082025-04-30T06:46:33.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
132092025-04-30T06:46:33.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ca3c3d6-efd2-45fa-a59f-b4c6371a79ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132102025-04-30T06:46:33.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
132112025-04-30T06:46:33.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc3afa5-db25-4e3c-b32d-cf23b4fa2b55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132122025-04-30T06:46:33.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
132132025-04-30T06:46:33.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b07773d-9b82-4f11-b80f-6ae94fb845c6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132142025-04-30T06:46:33.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
132152025-04-30T06:46:33.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "968b3a2c-a07f-442c-82ca-68ecf1d9ceeb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132162025-04-30T06:46:33.943ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
132172025-04-30T06:46:33.946ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c24368c-00bd-469a-b68f-93ed9a96c4a3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132182025-04-30T06:46:33.946ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
132192025-04-30T06:46:33.947ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132202025-04-30T06:46:33.947ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132212025-04-30T06:46:33.947ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132222025-04-30T06:46:33.947ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
132232025-04-30T06:46:33.948ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132242025-04-30T06:46:33.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "619e255f-cfe8-4a0a-8e0a-4805a32ea1af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132252025-04-30T06:46:33.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
132262025-04-30T06:46:33.952ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132272025-04-30T06:46:33.961ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132282025-04-30T06:46:33.961ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132292025-04-30T06:46:33.961ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
132302025-04-30T06:46:33.962ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132312025-04-30T06:46:33.965ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132322025-04-30T06:46:33.965ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
132332025-04-30T06:46:33.965ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132342025-04-30T06:46:33.970ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acd259ba-b7c8-49d0-9d44-60c994cba3e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132352025-04-30T06:46:33.971ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
132362025-04-30T06:46:33.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dd91d8d-948a-44ec-8d99-b32569789028", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132372025-04-30T06:46:33.977ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
132382025-04-30T06:46:33.979ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132392025-04-30T06:46:33.979ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132402025-04-30T06:46:33.979ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
132412025-04-30T06:46:33.980ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
132422025-04-30T06:46:33.985ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
132432025-04-30T06:46:33.985ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
132442025-04-30T06:46:33.988ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09edcf9-f16d-43d7-a030-c4f09f632aff", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132452025-04-30T06:46:33.989ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
132462025-04-30T06:46:33.989ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
132472025-04-30T06:46:33.989ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132482025-04-30T06:46:33.992ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "49e8ca8f-396e-4bdb-b7a2-caa8b6c02348", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132492025-04-30T06:46:33.992ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
132502025-04-30T06:46:33.996ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f06abc64-29f6-484b-85e6-03a5dee499ee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132512025-04-30T06:46:33.996ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
132522025-04-30T06:46:33.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3529f646-aac8-45c7-8482-7c225a06bc35", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132532025-04-30T06:46:33.997ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
132542025-04-30T06:46:34.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f26107-76db-4b7e-bf29-761eeada0619", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132552025-04-30T06:46:34.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
132562025-04-30T06:46:34.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf8d2a8-2d53-4c32-94cf-bf4152bcd7fa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132572025-04-30T06:46:34.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
132582025-04-30T06:46:34.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e7b4bbd-8f7e-44ff-99d1-c9378433561f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132592025-04-30T06:46:34.005ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
132602025-04-30T06:46:34.009ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
132612025-04-30T06:46:34.012ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7800ac5d-50d0-4119-b123-fe58fbbed121", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132622025-04-30T06:46:34.012ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
132632025-04-30T06:46:34.015ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fdabb23-7dda-4a9f-9f3b-7a3d3d33de16", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132642025-04-30T06:46:34.015ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
132652025-04-30T06:46:34.019ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a78d2088-46f9-4450-98aa-d35d42615677", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132662025-04-30T06:46:34.019ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
132672025-04-30T06:46:34.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7620cf58-d038-4e24-a3b3-4539b1cb2ef2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132682025-04-30T06:46:34.023ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
132692025-04-30T06:46:34.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7feb527-67d6-4ecd-8161-6b186fee5f85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132702025-04-30T06:46:34.027ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
132712025-04-30T06:46:34.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64cf877a-a757-444e-96fc-f9ea806d3525", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132722025-04-30T06:46:34.028ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
132732025-04-30T06:46:34.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19e9f823-a461-48de-ac34-b31e519fc959", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132742025-04-30T06:46:34.030ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
132752025-04-30T06:46:34.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84bd4d20-d3d4-444a-b332-0bc27bd81920", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132762025-04-30T06:46:34.032ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
132772025-04-30T06:46:34.034ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec82ff2-716a-4c17-a7d2-43563200278d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
132782025-04-30T06:46:34.034ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132792025-04-30T06:46:34.034ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132802025-04-30T06:46:34.034ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132812025-04-30T06:46:34.034ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132822025-04-30T06:46:34.035ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132832025-04-30T06:46:34.039ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132842025-04-30T06:46:34.049ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132852025-04-30T06:46:34.049ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
132862025-04-30T06:46:34.049ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132872025-04-30T06:46:34.049ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132882025-04-30T06:46:34.049ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132892025-04-30T06:46:34.050ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132902025-04-30T06:46:34.052ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132912025-04-30T06:46:34.058ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
132922025-04-30T06:46:34.058ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132932025-04-30T06:46:34.058ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132942025-04-30T06:46:34.059ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132952025-04-30T06:46:34.060ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132962025-04-30T06:46:34.062ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132972025-04-30T06:46:34.062ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132982025-04-30T06:46:34.062ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
132992025-04-30T06:46:34.063ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133002025-04-30T06:46:34.068ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133012025-04-30T06:46:34.086ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133022025-04-30T06:46:34.086ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133032025-04-30T06:46:34.086ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
133042025-04-30T06:46:34.087ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133052025-04-30T06:46:34.170ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
133062025-04-30T06:46:34.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
133072025-04-30T06:46:34.171ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:39476 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
133082025-04-30T06:46:34.173ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
133092025-04-30T06:46:34.173ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133102025-04-30T06:46:34.173ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
133112025-04-30T06:46:34.174ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133122025-04-30T06:46:34.174ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:39476 req_id = 76a6fa3b-9dae-4699-aab9-108c1e208fb2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133132025-04-30T06:46:34.175ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 746 local_addr = [::1]:54590 method = GET remote_addr = [::1]:39476 req_id = 76a6fa3b-9dae-4699-aab9-108c1e208fb2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133142025-04-30T06:46:34.175ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "76a6fa3b-9dae-4699-aab9-108c1e208fb2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133152025-04-30T06:46:34.177ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133162025-04-30T06:46:34.177ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133172025-04-30T06:46:34.177ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
133182025-04-30T06:46:34.182ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133192025-04-30T06:46:34.182ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133202025-04-30T06:46:34.182ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133212025-04-30T06:46:34.182ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133222025-04-30T06:46:34.183ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133232025-04-30T06:46:34.189ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133242025-04-30T06:46:34.197ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133252025-04-30T06:46:34.197ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133262025-04-30T06:46:34.197ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133272025-04-30T06:46:34.198ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133282025-04-30T06:46:34.239ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
133292025-04-30T06:46:34.240ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
133302025-04-30T06:46:34.240ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
133312025-04-30T06:46:34.240ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 461.043127ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
133322025-04-30T06:46:34.240ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
133332025-04-30T06:46:34.240ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133342025-04-30T06:46:34.240ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133352025-04-30T06:46:34.240ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133362025-04-30T06:46:34.240ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133372025-04-30T06:46:34.241ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133382025-04-30T06:46:34.244ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133392025-04-30T06:46:34.249ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133402025-04-30T06:46:34.249ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133412025-04-30T06:46:34.249ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133422025-04-30T06:46:34.250ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133432025-04-30T06:46:34.255ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
133442025-04-30T06:46:34.255ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
133452025-04-30T06:46:34.256ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 66 74 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ft..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54840 req_id = de7c744a-9cf4-4f0a-99a5-aea726e5edaa
133462025-04-30T06:46:34.256ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26228,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54840 req_id = de7c744a-9cf4-4f0a-99a5-aea726e5edaa
133472025-04-30T06:46:34.256ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
133482025-04-30T06:46:34.256ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133492025-04-30T06:46:34.256ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54840 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = de7c744a-9cf4-4f0a-99a5-aea726e5edaa
133502025-04-30T06:46:34.257ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4459, tv_nsec: 56272513 } })
133512025-04-30T06:46:34.257ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
133522025-04-30T06:46:34.257ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8c bf 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58900 req_id = abbb005d-7399-44e2-a93e-06e4b39cf026
133532025-04-30T06:46:34.258ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36031,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58900 req_id = abbb005d-7399-44e2-a93e-06e4b39cf026
133542025-04-30T06:46:34.258ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133552025-04-30T06:46:34.258ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58900 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = abbb005d-7399-44e2-a93e-06e4b39cf026
133562025-04-30T06:46:34.317ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
133572025-04-30T06:46:34.318ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133582025-04-30T06:46:34.318ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133592025-04-30T06:46:34.318ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133602025-04-30T06:46:34.319ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
133612025-04-30T06:46:34.319ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133622025-04-30T06:46:34.320ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 63 93 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c c............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62336 req_id = d49a3bb9-4228-4685-8abf-0851bcd95968
133632025-04-30T06:46:34.320ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25491,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62336 req_id = d49a3bb9-4228-4685-8abf-0851bcd95968
133642025-04-30T06:46:34.320ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
133652025-04-30T06:46:34.321ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25491, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62336 req_id = d49a3bb9-4228-4685-8abf-0851bcd95968
133662025-04-30T06:46:34.321ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
133672025-04-30T06:46:34.321ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
133682025-04-30T06:46:34.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
133692025-04-30T06:46:34.323ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133702025-04-30T06:46:34.333ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133712025-04-30T06:46:34.333ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133722025-04-30T06:46:34.333ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
133732025-04-30T06:46:34.333ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133742025-04-30T06:46:34.339ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8330954-8f10-47bf-9ed1-fc46ee9186a0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133752025-04-30T06:46:34.339ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
133762025-04-30T06:46:34.339ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133772025-04-30T06:46:34.339ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
133782025-04-30T06:46:34.340ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133792025-04-30T06:46:34.344ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d725f2e0-964b-4219-8092-63c9bf6594d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133802025-04-30T06:46:34.344ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
133812025-04-30T06:46:34.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52faba1f-111a-4521-bb86-24cb7bcf1035", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133822025-04-30T06:46:34.350ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
133832025-04-30T06:46:34.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39a39ce3-719e-4416-bb5a-07f23e31a636", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133842025-04-30T06:46:34.353ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
133852025-04-30T06:46:34.355ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb1ee0bf-7302-408a-974a-e985e2d59ec4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133862025-04-30T06:46:34.356ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
133872025-04-30T06:46:34.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc87dc2c-c5f1-4c88-b1b8-976debcd882d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133882025-04-30T06:46:34.361ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
133892025-04-30T06:46:34.364ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11204035-16c7-481b-a2fc-dda51d69c803", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133902025-04-30T06:46:34.364ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
133912025-04-30T06:46:34.371ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
133922025-04-30T06:46:34.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fedcadd-e973-4081-ac47-cfb05f501fb1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133932025-04-30T06:46:34.372ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
133942025-04-30T06:46:34.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc23cc61-bc50-43d4-be1b-d04d6e4cc06b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133952025-04-30T06:46:34.374ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
133962025-04-30T06:46:34.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bda91b29-b7a5-4b73-8a89-09aee48a5b14", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133972025-04-30T06:46:34.376ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
133982025-04-30T06:46:34.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b1de87-c673-453f-9991-8438aa95f4cc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
133992025-04-30T06:46:34.378ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
134002025-04-30T06:46:34.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38142394-4d79-4072-9090-a512e18e62b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134012025-04-30T06:46:34.380ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
134022025-04-30T06:46:34.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c225579-4670-4164-a8fd-abd02ac04bf6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134032025-04-30T06:46:34.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
134042025-04-30T06:46:34.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dbbef81-8517-42a4-a89c-93e509aa40ca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134052025-04-30T06:46:34.386ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
134062025-04-30T06:46:34.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a294265-e43c-444c-a37d-44eb23ef55ac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134072025-04-30T06:46:34.392ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
134082025-04-30T06:46:34.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "692c1e01-889b-4c31-b8f4-6235e8270340", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134092025-04-30T06:46:34.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
134102025-04-30T06:46:34.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f967cfb7-5c67-4f4f-903a-bc49de7ccb07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134112025-04-30T06:46:34.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
134122025-04-30T06:46:34.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "225ed7a7-5ce3-49b0-b443-53f0e448a0f1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134132025-04-30T06:46:34.402ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
134142025-04-30T06:46:34.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5053dd3b-85cc-4e02-af7a-d5052e47b3d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134152025-04-30T06:46:34.406ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
134162025-04-30T06:46:34.426ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134172025-04-30T06:46:34.426ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134182025-04-30T06:46:34.426ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134192025-04-30T06:46:34.426ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
134202025-04-30T06:46:34.427ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134212025-04-30T06:46:34.429ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb970c8-6b90-4f28-a66c-e5326aefcfaa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134222025-04-30T06:46:34.430ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
134232025-04-30T06:46:34.478ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134242025-04-30T06:46:34.486ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134252025-04-30T06:46:34.486ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134262025-04-30T06:46:34.486ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
134272025-04-30T06:46:34.487ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134282025-04-30T06:46:34.494ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7941c89b-54d6-458e-a807-c5a9fb9e9399", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134292025-04-30T06:46:34.494ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
134302025-04-30T06:46:34.497ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134312025-04-30T06:46:34.497ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
134322025-04-30T06:46:34.498ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134332025-04-30T06:46:34.505ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4e0e303-26d3-43f7-947e-96ae5c51cfb1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134342025-04-30T06:46:34.508ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
134352025-04-30T06:46:34.521ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6e39a7d-f2d6-4791-873c-0b85233f57b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134362025-04-30T06:46:34.522ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
134372025-04-30T06:46:34.527ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b05c28-83b6-420c-9c99-16f95bf8014f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134382025-04-30T06:46:34.527ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
134392025-04-30T06:46:34.532ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c8c9fe4-3a81-4c1c-9c3d-45bb69b62fd0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134402025-04-30T06:46:34.532ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
134412025-04-30T06:46:34.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ac2bec-4a76-4e71-a8f4-6e71d7bd3548", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134422025-04-30T06:46:34.534ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
134432025-04-30T06:46:34.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f4a80a-0c56-42a8-9975-cd7f51128238", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134442025-04-30T06:46:34.538ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
134452025-04-30T06:46:34.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6afa788b-91ce-4661-a9e1-e6feb46426f1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134462025-04-30T06:46:34.544ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
134472025-04-30T06:46:34.545ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
134482025-04-30T06:46:34.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd6478ba-73ad-4c66-a090-2604bd8d3056", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134492025-04-30T06:46:34.547ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
134502025-04-30T06:46:34.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b2eacef-5bbb-4abd-9e91-12cfeeef3a5d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134512025-04-30T06:46:34.549ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
134522025-04-30T06:46:34.550ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "628d51ee-2397-48f7-9552-eb4dcaf516ee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134532025-04-30T06:46:34.550ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
134542025-04-30T06:46:34.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8845352d-2c09-4414-885a-7ac4cf2e065d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134552025-04-30T06:46:34.552ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
134562025-04-30T06:46:34.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62ba826d-b5a9-46c3-97f9-7a011a1d44eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134572025-04-30T06:46:34.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
134582025-04-30T06:46:34.557ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf2107db-8e28-4981-9716-0f35fbb07804", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134592025-04-30T06:46:34.559ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
134602025-04-30T06:46:34.560ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e62c00b6-291f-4235-b9bf-4a7ba6b791a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134612025-04-30T06:46:34.561ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
134622025-04-30T06:46:34.563ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e173b44-4bbc-4224-ac26-198a174ad232", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134632025-04-30T06:46:34.564ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
134642025-04-30T06:46:34.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51cb369a-b815-49c8-b57a-b743fdfba327", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134652025-04-30T06:46:34.568ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
134662025-04-30T06:46:34.570ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88190db4-d8cd-4fbd-a2a3-c08349811f05", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134672025-04-30T06:46:34.572ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
134682025-04-30T06:46:34.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b19eb0c8-ca8f-4230-8431-fd18ab897cd3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134692025-04-30T06:46:34.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
134702025-04-30T06:46:34.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd9321fc-4a24-4484-af35-7456776c9e36", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134712025-04-30T06:46:34.576ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
134722025-04-30T06:46:34.580ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "216e0569-a5aa-4bf1-9cc3-38de9ffb81cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134732025-04-30T06:46:34.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
134742025-04-30T06:46:34.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d52cf6e-c3eb-4b9b-9a6b-021e7522424b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134752025-04-30T06:46:34.583ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
134762025-04-30T06:46:34.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c39c77-bddc-4e8f-8ecb-8a75b7a409c9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134772025-04-30T06:46:34.585ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
134782025-04-30T06:46:34.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a70c443-f8ab-4af3-a1db-5acada6beb19", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134792025-04-30T06:46:34.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
134802025-04-30T06:46:34.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c59f5a8-f3d3-4f4e-8f2d-bdc626876ac3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
134812025-04-30T06:46:34.589ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134822025-04-30T06:46:34.590ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134832025-04-30T06:46:34.590ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134842025-04-30T06:46:34.590ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134852025-04-30T06:46:34.591ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134862025-04-30T06:46:34.596ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134872025-04-30T06:46:34.602ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134882025-04-30T06:46:34.602ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134892025-04-30T06:46:34.602ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134902025-04-30T06:46:34.602ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134912025-04-30T06:46:34.602ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134922025-04-30T06:46:34.603ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134932025-04-30T06:46:34.606ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134942025-04-30T06:46:34.606ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134952025-04-30T06:46:34.606ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134962025-04-30T06:46:34.606ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
134972025-04-30T06:46:34.607ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134982025-04-30T06:46:34.609ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134992025-04-30T06:46:34.615ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135002025-04-30T06:46:34.615ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135012025-04-30T06:46:34.615ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135022025-04-30T06:46:34.616ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135032025-04-30T06:46:34.620ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135042025-04-30T06:46:34.626ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135052025-04-30T06:46:34.626ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135062025-04-30T06:46:34.626ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
135072025-04-30T06:46:34.626ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135082025-04-30T06:46:34.671ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
135092025-04-30T06:46:34.671ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
135102025-04-30T06:46:34.672ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:51799 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135112025-04-30T06:46:34.672ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135122025-04-30T06:46:34.672ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135132025-04-30T06:46:34.673ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:51799 req_id = 87833c2a-daf6-441e-a84d-4c7545a6051d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135142025-04-30T06:46:34.675ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 1622 local_addr = [::1]:54590 method = GET remote_addr = [::1]:51799 req_id = 87833c2a-daf6-441e-a84d-4c7545a6051d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135152025-04-30T06:46:34.675ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135162025-04-30T06:46:34.675ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
135172025-04-30T06:46:34.676ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135182025-04-30T06:46:34.676ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135192025-04-30T06:46:34.677ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135202025-04-30T06:46:34.677ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135212025-04-30T06:46:34.677ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135222025-04-30T06:46:34.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "87833c2a-daf6-441e-a84d-4c7545a6051d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:33 GMT"} })
135232025-04-30T06:46:34.679ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135242025-04-30T06:46:34.679ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135252025-04-30T06:46:34.679ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
135262025-04-30T06:46:34.681ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135272025-04-30T06:46:34.681ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135282025-04-30T06:46:34.681ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135292025-04-30T06:46:34.682ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135302025-04-30T06:46:34.683ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135312025-04-30T06:46:34.694ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135322025-04-30T06:46:34.710ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135332025-04-30T06:46:34.710ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135342025-04-30T06:46:34.710ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135352025-04-30T06:46:34.718ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135362025-04-30T06:46:34.770ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
135372025-04-30T06:46:34.770ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
135382025-04-30T06:46:34.770ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 530.351601ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
135392025-04-30T06:46:34.770ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
135402025-04-30T06:46:34.770ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135412025-04-30T06:46:34.771ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135422025-04-30T06:46:34.771ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135432025-04-30T06:46:34.771ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135442025-04-30T06:46:34.771ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135452025-04-30T06:46:34.772ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
135462025-04-30T06:46:34.774ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135472025-04-30T06:46:34.779ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135482025-04-30T06:46:34.779ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135492025-04-30T06:46:34.779ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135502025-04-30T06:46:34.779ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135512025-04-30T06:46:34.782ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
135522025-04-30T06:46:34.782ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
135532025-04-30T06:46:34.783ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 1a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 D............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63084 req_id = f8d24a26-5240-4895-a5c3-0c52c135b720
135542025-04-30T06:46:34.783ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63084 req_id = f8d24a26-5240-4895-a5c3-0c52c135b720
135552025-04-30T06:46:34.783ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
135562025-04-30T06:46:34.783ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135572025-04-30T06:46:34.784ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63084 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f8d24a26-5240-4895-a5c3-0c52c135b720
135582025-04-30T06:46:34.784ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4459, tv_nsec: 583538496 } })
135592025-04-30T06:46:34.784ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
135602025-04-30T06:46:34.784ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 01 f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56483 req_id = 3dbea274-322a-4f7d-beef-7611678d2f45
135612025-04-30T06:46:34.785ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 501,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56483 req_id = 3dbea274-322a-4f7d-beef-7611678d2f45
135622025-04-30T06:46:34.785ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135632025-04-30T06:46:34.785ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56483 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3dbea274-322a-4f7d-beef-7611678d2f45
135642025-04-30T06:46:34.839ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
135652025-04-30T06:46:34.840ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135662025-04-30T06:46:34.840ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135672025-04-30T06:46:34.840ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135682025-04-30T06:46:34.840ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
135692025-04-30T06:46:34.841ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135702025-04-30T06:46:34.841ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 32 28 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 2(..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52585 req_id = 1697007f-a8cd-4533-848f-5ec1875b60b5
135712025-04-30T06:46:34.841ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12840,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52585 req_id = 1697007f-a8cd-4533-848f-5ec1875b60b5
135722025-04-30T06:46:34.841ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
135732025-04-30T06:46:34.841ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12840, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52585 req_id = 1697007f-a8cd-4533-848f-5ec1875b60b5
135742025-04-30T06:46:34.842ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
135752025-04-30T06:46:34.842ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
135762025-04-30T06:46:34.842ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
135772025-04-30T06:46:34.844ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c97eca95-6b10-4b40-ba69-ea4a6df9769e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135782025-04-30T06:46:34.844ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
135792025-04-30T06:46:34.846ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6bee218-ea43-4d18-b263-4ce1d0b15959", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135802025-04-30T06:46:34.846ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
135812025-04-30T06:46:34.846ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135822025-04-30T06:46:34.855ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135832025-04-30T06:46:34.855ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135842025-04-30T06:46:34.855ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
135852025-04-30T06:46:34.856ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135862025-04-30T06:46:34.859ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135872025-04-30T06:46:34.859ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
135882025-04-30T06:46:34.859ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135892025-04-30T06:46:34.861ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa826ecc-1f6b-42d0-b40f-1801cdf8d9a8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135902025-04-30T06:46:34.861ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
135912025-04-30T06:46:34.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7b1935e-68b0-48d3-9579-dbdf86e372e4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135922025-04-30T06:46:34.867ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
135932025-04-30T06:46:34.869ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73564027-d958-4286-a4fe-ff4e62d2a851", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135942025-04-30T06:46:34.870ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
135952025-04-30T06:46:34.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ca371a-c767-4908-a9f0-3ce80700ebd9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135962025-04-30T06:46:34.873ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
135972025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b5536b4-abe5-4e1f-8b77-858ad72b0b43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
135982025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
135992025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ae1a53b-e354-47b0-bad3-367e9dd8e166", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136002025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
136012025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc44ce21-13e1-4301-9040-05ca25d1675d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136022025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
136032025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36c911a0-18dd-463d-8e4e-9bd6ea62c884", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136042025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
136052025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b2cd71b-99c4-4c01-98fd-1265d397d83d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136062025-04-30T06:46:34.880ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
136072025-04-30T06:46:34.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a879bbaa-b610-429d-834d-39c26108ee1f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136082025-04-30T06:46:34.881ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
136092025-04-30T06:46:34.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1ae0c98-213d-4215-99ce-7f0eda3bc8f4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136102025-04-30T06:46:34.883ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
136112025-04-30T06:46:34.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a20f89eb-9559-4626-8eee-9d2d8c9deefa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136122025-04-30T06:46:34.885ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
136132025-04-30T06:46:34.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d779579-4d21-4ef8-a09f-5bcd9488acc1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136142025-04-30T06:46:34.889ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
136152025-04-30T06:46:34.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc7f7c7-fb58-4313-8191-a9762a4a584b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136162025-04-30T06:46:34.891ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
136172025-04-30T06:46:34.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec9b5fb5-fa44-45c7-90e5-32ab9f4a9a63", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136182025-04-30T06:46:34.894ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
136192025-04-30T06:46:34.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12d820b5-65dc-495f-b872-b2ee9144e295", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136202025-04-30T06:46:34.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
136212025-04-30T06:46:34.899ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6272f06-00b0-43fa-b5c9-4ffc66798b0a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136222025-04-30T06:46:34.900ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
136232025-04-30T06:46:34.904ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
136242025-04-30T06:46:34.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3646c50-bc86-4d21-ac85-7e71cfec9722", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136252025-04-30T06:46:34.905ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
136262025-04-30T06:46:34.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09f537d0-bde9-413e-80cf-550b27b791a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136272025-04-30T06:46:34.908ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
136282025-04-30T06:46:34.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e69450-0050-43c6-9efc-4f20da0fdeef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136292025-04-30T06:46:34.909ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
136302025-04-30T06:46:34.910ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecb210fe-c368-4755-bce3-9586daf3ee94", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136312025-04-30T06:46:34.911ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
136322025-04-30T06:46:34.913ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4ba28e5-d6b5-4d5e-9e47-bda7ac4650bc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136332025-04-30T06:46:34.914ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
136342025-04-30T06:46:34.918ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6dda332-95b7-4bc0-9874-05705b18b55c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136352025-04-30T06:46:34.919ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
136362025-04-30T06:46:34.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb5f72d-fa8a-4c13-9735-72979e45cd35", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136372025-04-30T06:46:34.921ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
136382025-04-30T06:46:34.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "831fe0a2-00b1-43ce-9eb3-f361bdbf3f96", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136392025-04-30T06:46:34.924ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
136402025-04-30T06:46:34.926ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ac011e-1b17-4b5f-b69d-03b2941c8cc3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136412025-04-30T06:46:34.926ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
136422025-04-30T06:46:34.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1402739e-8d43-4018-aaeb-9bb940395b7e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136432025-04-30T06:46:34.929ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
136442025-04-30T06:46:34.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25a28fd3-1deb-41cf-ad1e-84779bc0f951", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136452025-04-30T06:46:34.932ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
136462025-04-30T06:46:34.933ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "186847cc-6bfe-43b0-ae9b-a289e6708f64", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136472025-04-30T06:46:34.934ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
136482025-04-30T06:46:34.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b84879f3-0581-47f8-b0e1-58b52bece012", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136492025-04-30T06:46:34.935ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
136502025-04-30T06:46:34.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6da6413-b108-465b-8819-3fe4e091b2fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136512025-04-30T06:46:34.936ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
136522025-04-30T06:46:34.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cdb22f5-a408-4125-9537-ed057d051ebf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136532025-04-30T06:46:34.937ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
136542025-04-30T06:46:34.939ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abfad02-7db7-425a-a10d-2111edbd3917", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136552025-04-30T06:46:34.940ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
136562025-04-30T06:46:34.944ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e530dfb-99c4-40ae-ab17-89fc12b6d895", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136572025-04-30T06:46:34.944ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
136582025-04-30T06:46:34.947ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc875bb0-a348-44e7-a744-b8ac3d362720", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136592025-04-30T06:46:34.948ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
136602025-04-30T06:46:34.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a43a130-f59f-4307-a1d5-b8892f12bd6c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136612025-04-30T06:46:34.951ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
136622025-04-30T06:46:34.955ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136632025-04-30T06:46:34.955ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136642025-04-30T06:46:34.955ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136652025-04-30T06:46:34.955ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
136662025-04-30T06:46:34.957ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136672025-04-30T06:46:34.957ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e82c85-2137-4f2a-9a4f-aa05ddf6d8cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136682025-04-30T06:46:34.957ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
136692025-04-30T06:46:34.960ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136702025-04-30T06:46:34.968ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136712025-04-30T06:46:34.968ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136722025-04-30T06:46:34.968ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
136732025-04-30T06:46:34.970ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136742025-04-30T06:46:34.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "425e2409-1817-43b2-8163-4c9c50aac19e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136752025-04-30T06:46:34.975ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
136762025-04-30T06:46:34.975ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136772025-04-30T06:46:34.975ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
136782025-04-30T06:46:34.976ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136792025-04-30T06:46:34.980ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136802025-04-30T06:46:34.980ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136812025-04-30T06:46:34.980ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
136822025-04-30T06:46:34.982ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
136832025-04-30T06:46:34.982ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "925af56a-269f-4b81-a35c-295950c442d2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136842025-04-30T06:46:34.983ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
136852025-04-30T06:46:34.986ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
136862025-04-30T06:46:34.986ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
136872025-04-30T06:46:34.990ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81772449-315f-4239-9e6d-5f800b4d04ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136882025-04-30T06:46:34.990ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
136892025-04-30T06:46:34.994ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
136902025-04-30T06:46:34.994ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136912025-04-30T06:46:34.995ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01f8ad4-3f73-41e7-930d-34803cd22a40", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136922025-04-30T06:46:34.995ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
136932025-04-30T06:46:34.999ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c34597-8a5b-4263-8e31-f893f361cedc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136942025-04-30T06:46:34.999ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
136952025-04-30T06:46:35.002ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "018492b0-9508-4c2d-8c4b-06380ef4fdc6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
136962025-04-30T06:46:35.002ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136972025-04-30T06:46:35.002ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136982025-04-30T06:46:35.002ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136992025-04-30T06:46:35.002ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137002025-04-30T06:46:35.003ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137012025-04-30T06:46:35.011ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137022025-04-30T06:46:35.019ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137032025-04-30T06:46:35.019ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137042025-04-30T06:46:35.019ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137052025-04-30T06:46:35.019ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137062025-04-30T06:46:35.019ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137072025-04-30T06:46:35.020ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137082025-04-30T06:46:35.024ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137092025-04-30T06:46:35.030ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137102025-04-30T06:46:35.030ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137112025-04-30T06:46:35.030ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137122025-04-30T06:46:35.031ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137132025-04-30T06:46:35.094ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
137142025-04-30T06:46:35.094ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
137152025-04-30T06:46:35.099ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:38064 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137162025-04-30T06:46:35.099ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
137172025-04-30T06:46:35.100ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:38064 req_id = a987fa8a-1a6c-4fb2-a9c6-892bb18e195a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137182025-04-30T06:46:35.101ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 963 local_addr = [::1]:54590 method = GET remote_addr = [::1]:38064 req_id = a987fa8a-1a6c-4fb2-a9c6-892bb18e195a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137192025-04-30T06:46:35.101ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
137202025-04-30T06:46:35.101ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a987fa8a-1a6c-4fb2-a9c6-892bb18e195a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137212025-04-30T06:46:35.103ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137222025-04-30T06:46:35.103ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137232025-04-30T06:46:35.103ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
137242025-04-30T06:46:35.104ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137252025-04-30T06:46:35.104ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137262025-04-30T06:46:35.104ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137272025-04-30T06:46:35.104ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137282025-04-30T06:46:35.105ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137292025-04-30T06:46:35.109ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137302025-04-30T06:46:35.118ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137312025-04-30T06:46:35.118ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137322025-04-30T06:46:35.118ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137332025-04-30T06:46:35.120ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137342025-04-30T06:46:35.152ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137352025-04-30T06:46:35.152ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137362025-04-30T06:46:35.152ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137372025-04-30T06:46:35.152ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
137382025-04-30T06:46:35.154ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137392025-04-30T06:46:35.160ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
137402025-04-30T06:46:35.161ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
137412025-04-30T06:46:35.161ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 390.533733ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
137422025-04-30T06:46:35.161ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
137432025-04-30T06:46:35.161ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137442025-04-30T06:46:35.161ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137452025-04-30T06:46:35.161ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137462025-04-30T06:46:35.161ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137472025-04-30T06:46:35.162ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137482025-04-30T06:46:35.162ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137492025-04-30T06:46:35.179ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137502025-04-30T06:46:35.179ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137512025-04-30T06:46:35.180ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
137522025-04-30T06:46:35.180ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137532025-04-30T06:46:35.182ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137542025-04-30T06:46:35.189ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137552025-04-30T06:46:35.189ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137562025-04-30T06:46:35.189ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137572025-04-30T06:46:35.189ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137582025-04-30T06:46:35.190ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137592025-04-30T06:46:35.190ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
137602025-04-30T06:46:35.191ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137612025-04-30T06:46:35.194ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
137622025-04-30T06:46:35.194ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
137632025-04-30T06:46:35.195ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2a 4d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 *M..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51298 req_id = 90253f44-7a2e-4f83-89f7-80983ffe3ecb
137642025-04-30T06:46:35.196ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10829,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51298 req_id = 90253f44-7a2e-4f83-89f7-80983ffe3ecb
137652025-04-30T06:46:35.196ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
137662025-04-30T06:46:35.196ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137672025-04-30T06:46:35.196ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51298 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 90253f44-7a2e-4f83-89f7-80983ffe3ecb
137682025-04-30T06:46:35.198ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4459, tv_nsec: 997249548 } })
137692025-04-30T06:46:35.198ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
137702025-04-30T06:46:35.198ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c c5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54326 req_id = 4a00d3b5-fd48-433a-bdb3-0b81cbfe1db5
137712025-04-30T06:46:35.199ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54326 req_id = 4a00d3b5-fd48-433a-bdb3-0b81cbfe1db5
137722025-04-30T06:46:35.199ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137732025-04-30T06:46:35.199ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54326 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4a00d3b5-fd48-433a-bdb3-0b81cbfe1db5
137742025-04-30T06:46:35.259ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
137752025-04-30T06:46:35.259ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6a 9f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c j............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59224 req_id = d7729a4f-f506-4ad5-8d0d-f27badc9b0fc
137762025-04-30T06:46:35.260ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59224 req_id = d7729a4f-f506-4ad5-8d0d-f27badc9b0fc
137772025-04-30T06:46:35.260ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
137782025-04-30T06:46:35.260ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27295, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59224 req_id = d7729a4f-f506-4ad5-8d0d-f27badc9b0fc
137792025-04-30T06:46:35.260ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
137802025-04-30T06:46:35.260ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
137812025-04-30T06:46:35.260ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
137822025-04-30T06:46:35.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cf73365-1c28-4ec1-a3f4-bf514e98f817", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137832025-04-30T06:46:35.265ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
137842025-04-30T06:46:35.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3aae77c-2a21-4eaa-9c22-69fc976b48cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137852025-04-30T06:46:35.268ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
137862025-04-30T06:46:35.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dfec67c-6cca-4210-802f-f77ad54ff518", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137872025-04-30T06:46:35.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
137882025-04-30T06:46:35.274ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ddf7353-0dc6-4f74-b0d3-31c135dece6e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137892025-04-30T06:46:35.274ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
137902025-04-30T06:46:35.277ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfa2b5d-000e-4697-996c-64a27e57ddf4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137912025-04-30T06:46:35.277ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
137922025-04-30T06:46:35.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74f575c5-3f2a-4223-ae06-3dfe4f89b29c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137932025-04-30T06:46:35.282ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
137942025-04-30T06:46:35.283ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
137952025-04-30T06:46:35.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "105624be-bba1-4a56-b17f-a4c80f1a4d3d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137962025-04-30T06:46:35.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
137972025-04-30T06:46:35.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef501101-590a-47f3-9868-b7d9849b5f45", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
137982025-04-30T06:46:35.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
137992025-04-30T06:46:35.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1707a29b-87fd-45da-a5bf-49535cc36114", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138002025-04-30T06:46:35.291ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
138012025-04-30T06:46:35.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e1bb3ef-a09f-44da-9dc3-e4c2e0a12c92", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138022025-04-30T06:46:35.293ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
138032025-04-30T06:46:35.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26f8b663-8418-4939-b785-b7177b2a195f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138042025-04-30T06:46:35.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
138052025-04-30T06:46:35.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "509aa83c-67c2-460a-8f85-76510548ea59", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138062025-04-30T06:46:35.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
138072025-04-30T06:46:35.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9292ae98-f7bb-4008-aa7d-110d3124af8a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138082025-04-30T06:46:35.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
138092025-04-30T06:46:35.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "828f905e-027e-4574-8059-7d3584c00fda", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138102025-04-30T06:46:35.301ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
138112025-04-30T06:46:35.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9423be09-9c97-41fc-a209-7c990a0ebdca", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138122025-04-30T06:46:35.303ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
138132025-04-30T06:46:35.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbdc8cb4-53d0-4cc4-ab9e-a0370972027a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138142025-04-30T06:46:35.307ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
138152025-04-30T06:46:35.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a4b45fd-6db4-47f0-999f-49d793de0163", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138162025-04-30T06:46:35.311ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
138172025-04-30T06:46:35.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aa5cbe7-02c4-4e39-bec3-848c1a648a7d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138182025-04-30T06:46:35.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
138192025-04-30T06:46:35.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9fed15b-35aa-4d7a-b2df-a26fee8f30fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138202025-04-30T06:46:35.317ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
138212025-04-30T06:46:35.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d53ebf-c2fe-439f-924d-16627017fa1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138222025-04-30T06:46:35.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
138232025-04-30T06:46:35.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44942f35-e3c5-4475-861a-d092a39e2dc3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138242025-04-30T06:46:35.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
138252025-04-30T06:46:35.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0379f26a-fecf-4a60-885f-658c67610996", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138262025-04-30T06:46:35.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
138272025-04-30T06:46:35.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74f7d6e-9bce-4bc9-903a-8b3dc9b35d67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138282025-04-30T06:46:35.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
138292025-04-30T06:46:35.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6af0f061-8473-4d13-9b48-be9eb3313363", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138302025-04-30T06:46:35.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
138312025-04-30T06:46:35.328ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c023b09-496d-4921-83a4-912c5eaeace0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138322025-04-30T06:46:35.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
138332025-04-30T06:46:35.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e43deb6-951a-46bc-82f1-59bb96774eb8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138342025-04-30T06:46:35.332ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
138352025-04-30T06:46:35.333ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138362025-04-30T06:46:35.333ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138372025-04-30T06:46:35.333ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138382025-04-30T06:46:35.333ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
138392025-04-30T06:46:35.335ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138402025-04-30T06:46:35.335ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e5cdf3-6f9f-4b65-bbc9-4eadba0bfbeb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138412025-04-30T06:46:35.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
138422025-04-30T06:46:35.337ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138432025-04-30T06:46:35.346ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138442025-04-30T06:46:35.346ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138452025-04-30T06:46:35.346ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
138462025-04-30T06:46:35.347ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138472025-04-30T06:46:35.350ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138482025-04-30T06:46:35.350ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
138492025-04-30T06:46:35.351ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138502025-04-30T06:46:35.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e26ddc-36f3-4d57-90ac-6c8470fe8860", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138512025-04-30T06:46:35.354ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
138522025-04-30T06:46:35.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e26aeb-5e17-4c36-99fc-59a92769ecb4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138532025-04-30T06:46:35.366ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
138542025-04-30T06:46:35.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc35d45-48a7-4478-901d-e49dac9ba13c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138552025-04-30T06:46:35.370ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
138562025-04-30T06:46:35.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b70b0d9b-c779-4c11-845b-44dba5e354c5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138572025-04-30T06:46:35.373ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
138582025-04-30T06:46:35.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab08f048-592b-4bad-bc33-d16c822ef875", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138592025-04-30T06:46:35.377ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
138602025-04-30T06:46:35.381ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df5a16f9-7cf5-4f7b-830e-d0f6de295e26", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138612025-04-30T06:46:35.382ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
138622025-04-30T06:46:35.384ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b18d337-8d5e-4a68-92d1-f954738bff85", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138632025-04-30T06:46:35.384ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
138642025-04-30T06:46:35.387ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a85756a-5e11-4367-84a5-2c57f2f0af28", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138652025-04-30T06:46:35.388ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
138662025-04-30T06:46:35.388ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
138672025-04-30T06:46:35.393ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4513ebfc-dfd8-4a02-9898-5d5df068660a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138682025-04-30T06:46:35.394ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
138692025-04-30T06:46:35.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d162aa46-540f-4b30-b427-dd6ab72ca2f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138702025-04-30T06:46:35.395ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
138712025-04-30T06:46:35.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "171e799a-8067-45a0-8d30-6acff59ec1c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138722025-04-30T06:46:35.397ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
138732025-04-30T06:46:35.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a5eee1-a67e-412f-a4fc-4fce1b1ec005", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138742025-04-30T06:46:35.399ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
138752025-04-30T06:46:35.403ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1c3e80a-ceb6-4649-af71-47a192a85d78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138762025-04-30T06:46:35.403ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
138772025-04-30T06:46:35.408ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32288d30-a177-4c4a-b375-cde5fc11cde8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138782025-04-30T06:46:35.408ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
138792025-04-30T06:46:35.422ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d234ad0d-e344-47e7-88c1-9ba0c4d99f1d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138802025-04-30T06:46:35.422ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
138812025-04-30T06:46:35.430ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b37a055e-3e90-4ff5-89b5-6afd80fe03e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138822025-04-30T06:46:35.430ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
138832025-04-30T06:46:35.435ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e41a963d-0a19-4611-a81f-6ef48270b362", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138842025-04-30T06:46:35.435ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
138852025-04-30T06:46:35.439ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9027c275-6cb6-471b-865a-9f44e706c32b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
138862025-04-30T06:46:35.439ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138872025-04-30T06:46:35.439ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138882025-04-30T06:46:35.439ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138892025-04-30T06:46:35.439ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138902025-04-30T06:46:35.440ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138912025-04-30T06:46:35.440ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138922025-04-30T06:46:35.440ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138932025-04-30T06:46:35.440ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138942025-04-30T06:46:35.440ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
138952025-04-30T06:46:35.441ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138962025-04-30T06:46:35.444ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138972025-04-30T06:46:35.448ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138982025-04-30T06:46:35.448ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
138992025-04-30T06:46:35.448ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139002025-04-30T06:46:35.448ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139012025-04-30T06:46:35.448ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139022025-04-30T06:46:35.449ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139032025-04-30T06:46:35.449ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139042025-04-30T06:46:35.456ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139052025-04-30T06:46:35.457ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139062025-04-30T06:46:35.457ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
139072025-04-30T06:46:35.457ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139082025-04-30T06:46:35.459ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139092025-04-30T06:46:35.466ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
139102025-04-30T06:46:35.466ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139112025-04-30T06:46:35.466ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139122025-04-30T06:46:35.467ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139132025-04-30T06:46:35.468ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139142025-04-30T06:46:35.468ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
139152025-04-30T06:46:35.468ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139162025-04-30T06:46:35.525ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
139172025-04-30T06:46:35.525ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
139182025-04-30T06:46:35.526ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:48381 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
139192025-04-30T06:46:35.526ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:48381 req_id = f16cea49-43b2-4991-91c0-55af16b2f83d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139202025-04-30T06:46:35.527ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 683 local_addr = [::1]:54590 method = GET remote_addr = [::1]:48381 req_id = f16cea49-43b2-4991-91c0-55af16b2f83d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
139212025-04-30T06:46:35.527ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f16cea49-43b2-4991-91c0-55af16b2f83d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
139222025-04-30T06:46:35.529ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139232025-04-30T06:46:35.529ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
139242025-04-30T06:46:35.529ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
139252025-04-30T06:46:35.531ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139262025-04-30T06:46:35.531ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139272025-04-30T06:46:35.531ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139282025-04-30T06:46:35.531ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139292025-04-30T06:46:35.532ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139302025-04-30T06:46:35.543ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139312025-04-30T06:46:35.548ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139322025-04-30T06:46:35.548ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139332025-04-30T06:46:35.548ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139342025-04-30T06:46:35.549ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139352025-04-30T06:46:35.550ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
139362025-04-30T06:46:35.550ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139372025-04-30T06:46:35.550ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 63958973-a70d-4ae2-98e8-837f00337c27 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139382025-04-30T06:46:35.551ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 63958973-a70d-4ae2-98e8-837f00337c27 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139392025-04-30T06:46:35.551ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 63958973-a70d-4ae2-98e8-837f00337c27 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139402025-04-30T06:46:35.551ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 63958973-a70d-4ae2-98e8-837f00337c27 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139412025-04-30T06:46:35.557ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6332 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 63958973-a70d-4ae2-98e8-837f00337c27 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
139422025-04-30T06:46:35.558ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "63958973-a70d-4ae2-98e8-837f00337c27", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
139432025-04-30T06:46:35.558ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139442025-04-30T06:46:35.559ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = c59dc491-90eb-444f-bca7-55d2cce8f03e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139452025-04-30T06:46:35.560ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = c59dc491-90eb-444f-bca7-55d2cce8f03e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139462025-04-30T06:46:35.560ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = c59dc491-90eb-444f-bca7-55d2cce8f03e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139472025-04-30T06:46:35.560ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = c59dc491-90eb-444f-bca7-55d2cce8f03e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139482025-04-30T06:46:35.568ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 8263 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = c59dc491-90eb-444f-bca7-55d2cce8f03e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
139492025-04-30T06:46:35.569ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c59dc491-90eb-444f-bca7-55d2cce8f03e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
139502025-04-30T06:46:35.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
139512025-04-30T06:46:35.569ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139522025-04-30T06:46:35.569ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
139532025-04-30T06:46:35.577ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
139542025-04-30T06:46:35.587ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
139552025-04-30T06:46:35.587ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
139562025-04-30T06:46:35.587ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 426.291358ms iteration = 52 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
139572025-04-30T06:46:35.587ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 53 reason = Signaled
139582025-04-30T06:46:35.587ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
139592025-04-30T06:46:35.587ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139602025-04-30T06:46:35.588ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139612025-04-30T06:46:35.588ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139622025-04-30T06:46:35.588ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139632025-04-30T06:46:35.595ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139642025-04-30T06:46:35.600ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
139652025-04-30T06:46:35.600ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
139662025-04-30T06:46:35.600ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
139672025-04-30T06:46:35.601ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
139682025-04-30T06:46:35.609ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
139692025-04-30T06:46:35.609ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
139702025-04-30T06:46:35.610ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fd dd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55215 req_id = 676008a0-baca-4ceb-a38f-220702c014dd
139712025-04-30T06:46:35.610ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64989,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55215 req_id = 676008a0-baca-4ceb-a38f-220702c014dd
139722025-04-30T06:46:35.611ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
139732025-04-30T06:46:35.611ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139742025-04-30T06:46:35.611ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55215 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 676008a0-baca-4ceb-a38f-220702c014dd
139752025-04-30T06:46:35.612ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4460, tv_nsec: 410928276 } })
139762025-04-30T06:46:35.612ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
139772025-04-30T06:46:35.612ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 16 5c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .\\..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63806 req_id = a290a8a4-03f0-4164-a031-0d8e7e0ee04b
139782025-04-30T06:46:35.612ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5724,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63806 req_id = a290a8a4-03f0-4164-a031-0d8e7e0ee04b
139792025-04-30T06:46:35.612ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139802025-04-30T06:46:35.613ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63806 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a290a8a4-03f0-4164-a031-0d8e7e0ee04b
139812025-04-30T06:46:35.667ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
139822025-04-30T06:46:35.667ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139832025-04-30T06:46:35.667ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139842025-04-30T06:46:35.667ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139852025-04-30T06:46:35.667ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
139862025-04-30T06:46:35.668ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139872025-04-30T06:46:35.668ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139882025-04-30T06:46:35.668ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139892025-04-30T06:46:35.670ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a0 15 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63504 req_id = df67e1cc-2594-4525-9459-5366451172ff
139902025-04-30T06:46:35.670ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40981,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63504 req_id = df67e1cc-2594-4525-9459-5366451172ff
139912025-04-30T06:46:35.670ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
139922025-04-30T06:46:35.670ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40981, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63504 req_id = df67e1cc-2594-4525-9459-5366451172ff
139932025-04-30T06:46:35.671ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
139942025-04-30T06:46:35.671ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
139952025-04-30T06:46:35.671ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
139962025-04-30T06:46:35.672ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139972025-04-30T06:46:35.672ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139982025-04-30T06:46:35.673ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139992025-04-30T06:46:35.673ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
140002025-04-30T06:46:35.673ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140012025-04-30T06:46:35.681ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140022025-04-30T06:46:35.681ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140032025-04-30T06:46:35.681ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
140042025-04-30T06:46:35.681ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140052025-04-30T06:46:35.683ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5431f7a2-c86c-4be2-8d46-4d3333b3bce8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140062025-04-30T06:46:35.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
140072025-04-30T06:46:35.684ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140082025-04-30T06:46:35.684ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
140092025-04-30T06:46:35.685ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140102025-04-30T06:46:35.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a8607e-0284-4f7b-8908-57d02adf6ef8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140112025-04-30T06:46:35.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
140122025-04-30T06:46:35.691ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d12230f3-6ec9-47e3-904b-ed8e39eab01d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140132025-04-30T06:46:35.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
140142025-04-30T06:46:35.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad09f56-cca6-4f6c-8a3b-860a4c314b7d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140152025-04-30T06:46:35.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
140162025-04-30T06:46:35.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42fe2d62-dc43-434f-9118-84ea27b92cbc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140172025-04-30T06:46:35.699ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
140182025-04-30T06:46:35.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a1f342-2853-44d6-ab07-392c0eb0d30f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140192025-04-30T06:46:35.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
140202025-04-30T06:46:35.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16eb45b2-eb14-426d-9e54-67aa57cd5f48", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140212025-04-30T06:46:35.704ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
140222025-04-30T06:46:35.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e265ab8-3e0d-4d91-b2d6-d70627ed477b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140232025-04-30T06:46:35.708ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
140242025-04-30T06:46:35.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93b81479-e64d-4e69-9423-5032adf20fd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140252025-04-30T06:46:35.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
140262025-04-30T06:46:35.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acc38267-1140-4776-9ad0-44452676c36c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140272025-04-30T06:46:35.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
140282025-04-30T06:46:35.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "107f8b2a-dabb-40d2-a7c2-288aa0603294", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140292025-04-30T06:46:35.713ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
140302025-04-30T06:46:35.713ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
140312025-04-30T06:46:35.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffb4a143-7d0b-49d8-9951-d5fd9963fb05", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140322025-04-30T06:46:35.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
140332025-04-30T06:46:35.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a0cc7e9-47f7-47fc-98df-dd57d4a519bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140342025-04-30T06:46:35.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
140352025-04-30T06:46:35.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e09698-6a4e-4a53-a60c-3e13c1eab92f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140362025-04-30T06:46:35.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
140372025-04-30T06:46:35.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c85685d-027f-4263-a315-b68c5da90cb4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140382025-04-30T06:46:35.721ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
140392025-04-30T06:46:35.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94033e16-2219-4cff-bd91-d77004498429", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140402025-04-30T06:46:35.732ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
140412025-04-30T06:46:35.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0073d4d8-e71d-4ad0-8520-3e730b68f544", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140422025-04-30T06:46:35.737ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
140432025-04-30T06:46:35.741ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e3ddbd-fac9-4312-af8d-5cc7d636e4b6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140442025-04-30T06:46:35.741ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
140452025-04-30T06:46:35.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7183a3e1-091f-4dc5-a692-9fd7569eab50", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140462025-04-30T06:46:35.756ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
140472025-04-30T06:46:35.758ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8846e5-b9a6-41b3-b380-4e4aa2881829", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140482025-04-30T06:46:35.758ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
140492025-04-30T06:46:35.760ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "089b6aab-b1c4-4eee-a99c-8cfff9805563", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140502025-04-30T06:46:35.760ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
140512025-04-30T06:46:35.762ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "067bb496-cbf9-4054-96eb-0e9794f83ff3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140522025-04-30T06:46:35.762ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
140532025-04-30T06:46:35.763ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "850dd03b-b62d-4ea5-b1b0-f5a22700d6d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140542025-04-30T06:46:35.764ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
140552025-04-30T06:46:35.764ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140562025-04-30T06:46:35.764ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140572025-04-30T06:46:35.764ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140582025-04-30T06:46:35.764ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
140592025-04-30T06:46:35.765ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140602025-04-30T06:46:35.769ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "071eb350-d716-4d68-bab9-b4dd5c8c1189", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140612025-04-30T06:46:35.769ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
140622025-04-30T06:46:35.771ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140632025-04-30T06:46:35.781ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140642025-04-30T06:46:35.781ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140652025-04-30T06:46:35.781ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
140662025-04-30T06:46:35.782ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140672025-04-30T06:46:35.788ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140682025-04-30T06:46:35.788ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
140692025-04-30T06:46:35.791ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140702025-04-30T06:46:35.793ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94cb6d1e-e1e9-4de8-8ac1-4ac25b22a965", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140712025-04-30T06:46:35.794ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
140722025-04-30T06:46:35.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee47cca6-1c93-4b9c-b4f1-82cb26b84851", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140732025-04-30T06:46:35.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
140742025-04-30T06:46:35.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99b8b856-4659-4a44-9a94-194abdb60562", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140752025-04-30T06:46:35.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
140762025-04-30T06:46:35.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24a95142-c92e-4192-b404-f7d9efb784d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140772025-04-30T06:46:35.803ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
140782025-04-30T06:46:35.807ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e949bd63-93b5-4f1e-9b0c-92b6745669df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140792025-04-30T06:46:35.807ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
140802025-04-30T06:46:35.809ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2964e810-7e81-4c5c-b0db-6cb9d5362d14", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140812025-04-30T06:46:35.810ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
140822025-04-30T06:46:35.811ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51d96797-f9c2-4a6f-b9f4-00fdc62beda7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140832025-04-30T06:46:35.811ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
140842025-04-30T06:46:35.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc7c89a-2d08-4ed8-8ffe-3f477575600e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140852025-04-30T06:46:35.814ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
140862025-04-30T06:46:35.818ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "208c94ec-21d4-4b61-97ce-56965adacfc4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140872025-04-30T06:46:35.819ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
140882025-04-30T06:46:35.821ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a58fb7f-91fa-4c69-8cde-49f1d5359351", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140892025-04-30T06:46:35.822ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
140902025-04-30T06:46:35.826ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
140912025-04-30T06:46:35.828ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "565e998b-761d-4aa6-8934-4dde7908b705", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140922025-04-30T06:46:35.829ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
140932025-04-30T06:46:35.831ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a34f0887-1034-4003-a15b-f99c0b95b5e1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140942025-04-30T06:46:35.831ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
140952025-04-30T06:46:35.834ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7a7aa6-1a73-4746-b19d-53de3d1e05ef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140962025-04-30T06:46:35.834ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
140972025-04-30T06:46:35.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "badd770e-ad51-4d98-81ec-080d1e1e630a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
140982025-04-30T06:46:35.837ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
140992025-04-30T06:46:35.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b23ec923-6108-447e-ab07-559cad3effe7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
141002025-04-30T06:46:35.839ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
141012025-04-30T06:46:35.842ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0588fea-e605-4c78-899d-e1f96d11bd0c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:34 GMT"} })
141022025-04-30T06:46:35.842ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
141032025-04-30T06:46:35.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec005981-c56a-42a8-933a-3e7da53427cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141042025-04-30T06:46:35.845ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
141052025-04-30T06:46:35.846ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "596f9a3b-4281-430e-b5a5-7ae3da7eb711", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141062025-04-30T06:46:35.846ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
141072025-04-30T06:46:35.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a20f4524-4a78-4828-9578-254013c0e548", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141082025-04-30T06:46:35.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
141092025-04-30T06:46:35.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f7388c9-a9a4-461c-9aaf-b4e7179a030b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141102025-04-30T06:46:35.848ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
141112025-04-30T06:46:35.849ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5882d56-3abd-40bd-9afe-eb41cae2b536", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141122025-04-30T06:46:35.849ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141132025-04-30T06:46:35.849ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141142025-04-30T06:46:35.849ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141152025-04-30T06:46:35.849ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141162025-04-30T06:46:35.850ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141172025-04-30T06:46:35.854ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141182025-04-30T06:46:35.862ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141192025-04-30T06:46:35.862ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
141202025-04-30T06:46:35.862ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141212025-04-30T06:46:35.862ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141222025-04-30T06:46:35.862ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141232025-04-30T06:46:35.863ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141242025-04-30T06:46:35.867ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141252025-04-30T06:46:35.875ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
141262025-04-30T06:46:35.876ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141272025-04-30T06:46:35.876ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141282025-04-30T06:46:35.876ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141292025-04-30T06:46:35.878ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141302025-04-30T06:46:35.878ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141312025-04-30T06:46:35.878ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141322025-04-30T06:46:35.878ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
141332025-04-30T06:46:35.878ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141342025-04-30T06:46:35.882ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141352025-04-30T06:46:35.888ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141362025-04-30T06:46:35.888ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141372025-04-30T06:46:35.889ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
141382025-04-30T06:46:35.890ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141392025-04-30T06:46:35.961ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
141402025-04-30T06:46:35.961ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
141412025-04-30T06:46:35.962ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:64721 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
141422025-04-30T06:46:35.962ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:64721 req_id = 356af6f9-3036-4199-87fa-6fc8c0029f69 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141432025-04-30T06:46:35.963ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 677 local_addr = [::1]:54590 method = GET remote_addr = [::1]:64721 req_id = 356af6f9-3036-4199-87fa-6fc8c0029f69 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141442025-04-30T06:46:35.964ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141452025-04-30T06:46:35.964ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
141462025-04-30T06:46:35.965ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141472025-04-30T06:46:35.966ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "356af6f9-3036-4199-87fa-6fc8c0029f69", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
141482025-04-30T06:46:35.969ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141492025-04-30T06:46:35.969ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141502025-04-30T06:46:35.969ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
141512025-04-30T06:46:35.971ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141522025-04-30T06:46:35.971ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141532025-04-30T06:46:35.971ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141542025-04-30T06:46:35.971ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141552025-04-30T06:46:35.971ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141562025-04-30T06:46:35.976ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141572025-04-30T06:46:35.984ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141582025-04-30T06:46:35.984ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141592025-04-30T06:46:35.984ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141602025-04-30T06:46:35.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141612025-04-30T06:46:35.987ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141622025-04-30T06:46:35.988ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141632025-04-30T06:46:35.988ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
141642025-04-30T06:46:35.989ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
141652025-04-30T06:46:36.007ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
141662025-04-30T06:46:36.007ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
141672025-04-30T06:46:36.015ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
141682025-04-30T06:46:36.019ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
141692025-04-30T06:46:36.052ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
141702025-04-30T06:46:36.053ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
141712025-04-30T06:46:36.054ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
141722025-04-30T06:46:36.054ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 466.212998ms iteration = 53 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
141732025-04-30T06:46:36.054ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 54 reason = Signaled
141742025-04-30T06:46:36.054ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141752025-04-30T06:46:36.054ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141762025-04-30T06:46:36.055ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141772025-04-30T06:46:36.055ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141782025-04-30T06:46:36.057ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141792025-04-30T06:46:36.064ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141802025-04-30T06:46:36.069ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141812025-04-30T06:46:36.070ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141822025-04-30T06:46:36.070ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141832025-04-30T06:46:36.071ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141842025-04-30T06:46:36.085ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
141852025-04-30T06:46:36.085ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
141862025-04-30T06:46:36.087ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bf 00 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50709 req_id = 104a5d7f-49b2-4833-b372-9379c0c3df92
141872025-04-30T06:46:36.087ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48896,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50709 req_id = 104a5d7f-49b2-4833-b372-9379c0c3df92
141882025-04-30T06:46:36.087ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
141892025-04-30T06:46:36.087ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141902025-04-30T06:46:36.087ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50709 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 104a5d7f-49b2-4833-b372-9379c0c3df92
141912025-04-30T06:46:36.088ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4460, tv_nsec: 887054165 } })
141922025-04-30T06:46:36.088ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
141932025-04-30T06:46:36.088ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5e de 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ^...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63988 req_id = e1772921-0e6f-4f2f-afb7-6f48a54bd000
141942025-04-30T06:46:36.088ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24286,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63988 req_id = e1772921-0e6f-4f2f-afb7-6f48a54bd000
141952025-04-30T06:46:36.088ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141962025-04-30T06:46:36.088ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63988 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e1772921-0e6f-4f2f-afb7-6f48a54bd000
141972025-04-30T06:46:36.181ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
141982025-04-30T06:46:36.182ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141992025-04-30T06:46:36.182ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142002025-04-30T06:46:36.182ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142012025-04-30T06:46:36.182ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142022025-04-30T06:46:36.183ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142032025-04-30T06:46:36.184ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
142042025-04-30T06:46:36.184ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c6 0b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52063 req_id = b7958147-40a8-4ebd-866c-d11863f7970f
142052025-04-30T06:46:36.185ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50699,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52063 req_id = b7958147-40a8-4ebd-866c-d11863f7970f
142062025-04-30T06:46:36.185ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
142072025-04-30T06:46:36.185ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50699, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52063 req_id = b7958147-40a8-4ebd-866c-d11863f7970f
142082025-04-30T06:46:36.187ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
142092025-04-30T06:46:36.187ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
142102025-04-30T06:46:36.187ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
142112025-04-30T06:46:36.187ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142122025-04-30T06:46:36.193ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142132025-04-30T06:46:36.193ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142142025-04-30T06:46:36.193ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142152025-04-30T06:46:36.194ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142162025-04-30T06:46:36.197ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142172025-04-30T06:46:36.197ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142182025-04-30T06:46:36.200ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142192025-04-30T06:46:36.212ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb2c456-8f01-4077-9414-bcba9137e2c8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142202025-04-30T06:46:36.213ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
142212025-04-30T06:46:36.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d79b74e0-27bc-4394-9083-fa56568262ea", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142222025-04-30T06:46:36.215ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
142232025-04-30T06:46:36.217ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaf9f2ff-fd05-4af7-8e9d-4a682ba864ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142242025-04-30T06:46:36.218ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
142252025-04-30T06:46:36.229ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23b83ab8-b33a-4f45-b105-1a29917c3b16", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142262025-04-30T06:46:36.230ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
142272025-04-30T06:46:36.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0525eecd-e383-4ec8-8d81-ae7a34cc4c84", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142282025-04-30T06:46:36.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
142292025-04-30T06:46:36.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d65f365-9297-4ee7-b42d-6a6b280e5a24", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142302025-04-30T06:46:36.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
142312025-04-30T06:46:36.248ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
142322025-04-30T06:46:36.250ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced51604-0406-470e-817c-c7f86a70fbaf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142332025-04-30T06:46:36.250ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
142342025-04-30T06:46:36.255ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9beecd0-2bd1-4b91-a85b-6dfca736372d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142352025-04-30T06:46:36.256ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
142362025-04-30T06:46:36.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbbcfe24-f2a3-45ee-9bb6-dd0b731ddebe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142372025-04-30T06:46:36.257ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
142382025-04-30T06:46:36.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35674f12-5fba-46b4-8840-5836a75d3f19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142392025-04-30T06:46:36.259ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
142402025-04-30T06:46:36.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecc87fb2-00db-4825-9448-c8a381a6c1dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142412025-04-30T06:46:36.261ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
142422025-04-30T06:46:36.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb343f8-2ade-478d-8d3a-cf3d9c300c9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142432025-04-30T06:46:36.263ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
142442025-04-30T06:46:36.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9908885c-215b-45c6-80b9-ea4fe6f177cc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142452025-04-30T06:46:36.264ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
142462025-04-30T06:46:36.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc81ae9-dac0-42bd-b456-d840fab771a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142472025-04-30T06:46:36.267ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
142482025-04-30T06:46:36.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1992eaa9-1552-4fa2-a059-f2a2103291f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142492025-04-30T06:46:36.271ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
142502025-04-30T06:46:36.277ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ed3984f-8175-4889-87b8-596e1253342d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142512025-04-30T06:46:36.277ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
142522025-04-30T06:46:36.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dcd10a2-99d3-437e-8238-f221c8418210", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142532025-04-30T06:46:36.279ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
142542025-04-30T06:46:36.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad972fc2-0558-442d-a43b-b01a1df8a54d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142552025-04-30T06:46:36.283ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
142562025-04-30T06:46:36.285ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f26728-d6b5-45bc-8926-27acd858913a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142572025-04-30T06:46:36.287ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
142582025-04-30T06:46:36.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc9d30fd-90a4-449e-971e-3da4efade32e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142592025-04-30T06:46:36.290ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
142602025-04-30T06:46:36.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ad45ca6-5070-4399-9137-f994b9628c76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142612025-04-30T06:46:36.292ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
142622025-04-30T06:46:36.293ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a6e9a50-8317-4980-8874-f5f291c254cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142632025-04-30T06:46:36.293ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
142642025-04-30T06:46:36.294ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e88c1ab-433a-47a2-ac8c-9d913dd9c282", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142652025-04-30T06:46:36.295ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
142662025-04-30T06:46:36.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cc2d9c9-1f0a-440a-b142-67470321b606", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142672025-04-30T06:46:36.296ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
142682025-04-30T06:46:36.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0d250d9-10a6-4059-a107-f5abd15199bb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142692025-04-30T06:46:36.298ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
142702025-04-30T06:46:36.300ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142712025-04-30T06:46:36.300ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142722025-04-30T06:46:36.300ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142732025-04-30T06:46:36.300ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142742025-04-30T06:46:36.301ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142752025-04-30T06:46:36.303ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142762025-04-30T06:46:36.309ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142772025-04-30T06:46:36.309ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142782025-04-30T06:46:36.309ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142792025-04-30T06:46:36.309ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142802025-04-30T06:46:36.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dedf93c4-0c0c-475e-97bb-1760130433c3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142812025-04-30T06:46:36.310ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
142822025-04-30T06:46:36.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd89fc05-3d80-4c3f-a842-9f865aacd127", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142832025-04-30T06:46:36.313ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
142842025-04-30T06:46:36.313ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142852025-04-30T06:46:36.313ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
142862025-04-30T06:46:36.314ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142872025-04-30T06:46:36.316ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec61d6b2-13c5-4f38-ae1d-ff1eeddcf61e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142882025-04-30T06:46:36.316ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
142892025-04-30T06:46:36.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27707b36-899f-466a-aaa4-eb3d6c141f3c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142902025-04-30T06:46:36.318ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
142912025-04-30T06:46:36.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f78285-ff10-47c8-b259-d61cc2d3ffa1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142922025-04-30T06:46:36.321ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
142932025-04-30T06:46:36.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33291811-7c20-4653-a33f-2fd3be0ae1ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142942025-04-30T06:46:36.322ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
142952025-04-30T06:46:36.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81a0c665-d6e5-45a1-9079-cff9ca9e822f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142962025-04-30T06:46:36.324ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
142972025-04-30T06:46:36.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bc9f0fc-e8d0-4696-9080-40de6cff6302", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
142982025-04-30T06:46:36.325ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
142992025-04-30T06:46:36.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cac5fb08-fdaa-4cf7-802c-0030d78a125d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143002025-04-30T06:46:36.326ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
143012025-04-30T06:46:36.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb1f570-c255-4ffb-a63d-be23e5ea818b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143022025-04-30T06:46:36.329ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
143032025-04-30T06:46:36.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5949fc-3eea-45f1-8d37-0529ac9946ed", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143042025-04-30T06:46:36.331ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
143052025-04-30T06:46:36.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bddd6441-c1fd-4be4-adf1-8bdb031bde1e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143062025-04-30T06:46:36.333ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
143072025-04-30T06:46:36.336ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b25ee90-378b-4d7f-a046-38fbe5ff6611", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143082025-04-30T06:46:36.337ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
143092025-04-30T06:46:36.341ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee2f1ae3-bb71-4d63-aadf-0b326948408a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143102025-04-30T06:46:36.341ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
143112025-04-30T06:46:36.342ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
143122025-04-30T06:46:36.345ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3cdfbbc-1f7d-4283-9f65-dc5400000179", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143132025-04-30T06:46:36.345ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
143142025-04-30T06:46:36.347ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8469bceb-7645-4ede-bb89-a9bb24259adc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143152025-04-30T06:46:36.347ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
143162025-04-30T06:46:36.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02174d0c-fbf6-41b0-bfc6-a4167c26e75a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143172025-04-30T06:46:36.348ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
143182025-04-30T06:46:36.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23aa9070-92be-4d25-9efb-a2317ed42aa1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143192025-04-30T06:46:36.349ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
143202025-04-30T06:46:36.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ddc3709-e0be-4336-9acd-260cf0a5acfb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143212025-04-30T06:46:36.351ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
143222025-04-30T06:46:36.352ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ade97399-a49c-45a3-8b09-ef4f3cde65e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143232025-04-30T06:46:36.352ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143242025-04-30T06:46:36.352ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143252025-04-30T06:46:36.352ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143262025-04-30T06:46:36.352ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143272025-04-30T06:46:36.353ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143282025-04-30T06:46:36.355ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143292025-04-30T06:46:36.366ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143302025-04-30T06:46:36.366ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143312025-04-30T06:46:36.366ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143322025-04-30T06:46:36.366ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143332025-04-30T06:46:36.366ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143342025-04-30T06:46:36.368ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143352025-04-30T06:46:36.374ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143362025-04-30T06:46:36.381ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143372025-04-30T06:46:36.381ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143382025-04-30T06:46:36.381ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143392025-04-30T06:46:36.382ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143402025-04-30T06:46:36.430ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
143412025-04-30T06:46:36.430ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
143422025-04-30T06:46:36.430ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143432025-04-30T06:46:36.430ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143442025-04-30T06:46:36.430ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143452025-04-30T06:46:36.430ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
143462025-04-30T06:46:36.431ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143472025-04-30T06:46:36.431ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:45979 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143482025-04-30T06:46:36.433ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:45979 req_id = dc13cded-a6da-4aeb-bb8c-75b702f85bcd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143492025-04-30T06:46:36.433ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 782 local_addr = [::1]:54590 method = GET remote_addr = [::1]:45979 req_id = dc13cded-a6da-4aeb-bb8c-75b702f85bcd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143502025-04-30T06:46:36.435ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc13cded-a6da-4aeb-bb8c-75b702f85bcd", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
143512025-04-30T06:46:36.437ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143522025-04-30T06:46:36.437ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143532025-04-30T06:46:36.437ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
143542025-04-30T06:46:36.439ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143552025-04-30T06:46:36.439ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143562025-04-30T06:46:36.439ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143572025-04-30T06:46:36.439ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143582025-04-30T06:46:36.439ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143592025-04-30T06:46:36.440ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143602025-04-30T06:46:36.447ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143612025-04-30T06:46:36.447ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143622025-04-30T06:46:36.447ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
143632025-04-30T06:46:36.448ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143642025-04-30T06:46:36.461ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143652025-04-30T06:46:36.462ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143662025-04-30T06:46:36.462ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143672025-04-30T06:46:36.462ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143682025-04-30T06:46:36.462ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143692025-04-30T06:46:36.466ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143702025-04-30T06:46:36.466ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
143712025-04-30T06:46:36.466ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143722025-04-30T06:46:36.489ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:55624 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143732025-04-30T06:46:36.490ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:55624 req_id = 162db6c3-548a-47c1-a401-c8607199eca8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143742025-04-30T06:46:36.490ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 766 local_addr = [::1]:64080 method = GET remote_addr = [::1]:55624 req_id = 162db6c3-548a-47c1-a401-c8607199eca8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
143752025-04-30T06:46:36.520ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
143762025-04-30T06:46:36.521ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
143772025-04-30T06:46:36.521ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 465.37521ms iteration = 54 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
143782025-04-30T06:46:36.521ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 55 reason = Signaled
143792025-04-30T06:46:36.521ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143802025-04-30T06:46:36.521ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143812025-04-30T06:46:36.521ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143822025-04-30T06:46:36.521ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143832025-04-30T06:46:36.522ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143842025-04-30T06:46:36.527ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143852025-04-30T06:46:36.534ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143862025-04-30T06:46:36.534ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143872025-04-30T06:46:36.534ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143882025-04-30T06:46:36.534ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143892025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
143902025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
143912025-04-30T06:46:36.540ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
143922025-04-30T06:46:36.540ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b7 66 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .f..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50472 req_id = aa4d2898-3c89-46ee-86e5-77c0ea66f2b3
143932025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46950,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50472 req_id = aa4d2898-3c89-46ee-86e5-77c0ea66f2b3
143942025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
143952025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143962025-04-30T06:46:36.540ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50472 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa4d2898-3c89-46ee-86e5-77c0ea66f2b3
143972025-04-30T06:46:36.541ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4461, tv_nsec: 339969079 } })
143982025-04-30T06:46:36.541ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
143992025-04-30T06:46:36.542ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2d c2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e -...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63411 req_id = 8f855d27-cf80-43d3-9960-4d4a0d962bc7
144002025-04-30T06:46:36.542ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11714,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63411 req_id = 8f855d27-cf80-43d3-9960-4d4a0d962bc7
144012025-04-30T06:46:36.542ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144022025-04-30T06:46:36.543ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63411 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8f855d27-cf80-43d3-9960-4d4a0d962bc7
144032025-04-30T06:46:36.617ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
144042025-04-30T06:46:36.617ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
144052025-04-30T06:46:36.618ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144062025-04-30T06:46:36.618ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144072025-04-30T06:46:36.618ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
144082025-04-30T06:46:36.618ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144092025-04-30T06:46:36.619ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4f 77 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Ow..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62559 req_id = 51f4e100-ac3f-49f5-9efa-bc0c781c7348
144102025-04-30T06:46:36.619ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20343,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62559 req_id = 51f4e100-ac3f-49f5-9efa-bc0c781c7348
144112025-04-30T06:46:36.619ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
144122025-04-30T06:46:36.620ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20343, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62559 req_id = 51f4e100-ac3f-49f5-9efa-bc0c781c7348
144132025-04-30T06:46:36.620ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
144142025-04-30T06:46:36.620ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
144152025-04-30T06:46:36.620ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
144162025-04-30T06:46:36.623ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144172025-04-30T06:46:36.630ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144182025-04-30T06:46:36.630ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144192025-04-30T06:46:36.630ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
144202025-04-30T06:46:36.630ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144212025-04-30T06:46:36.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e6d4748-9809-48a0-83bf-bca16b453f9a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144222025-04-30T06:46:36.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
144232025-04-30T06:46:36.635ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae952757-c1b2-4f4e-9a5e-704b27ae6e6b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144242025-04-30T06:46:36.635ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
144252025-04-30T06:46:36.635ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144262025-04-30T06:46:36.635ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
144272025-04-30T06:46:36.636ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144282025-04-30T06:46:36.636ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144292025-04-30T06:46:36.636ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144302025-04-30T06:46:36.640ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144312025-04-30T06:46:36.640ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144322025-04-30T06:46:36.641ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144332025-04-30T06:46:36.641ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
144342025-04-30T06:46:36.641ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f2adba-9b42-4b82-af73-00d859fb24e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144352025-04-30T06:46:36.642ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
144362025-04-30T06:46:36.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c8bc399-fd95-42d0-acd8-1d8a5e17db60", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144372025-04-30T06:46:36.646ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
144382025-04-30T06:46:36.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04f24e9e-8035-414a-952d-77533a6342f8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144392025-04-30T06:46:36.647ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
144402025-04-30T06:46:36.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "603798cb-4b59-4f51-ae77-af46585169ac", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144412025-04-30T06:46:36.650ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
144422025-04-30T06:46:36.652ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14a0f4b7-9f2a-4fbe-b886-7afcd395fd6a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144432025-04-30T06:46:36.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
144442025-04-30T06:46:36.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee7491b2-43d7-4364-b0fa-a8f53f7ab301", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144452025-04-30T06:46:36.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
144462025-04-30T06:46:36.659ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c953f8c7-46ea-470b-8608-5426e53b2703", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144472025-04-30T06:46:36.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
144482025-04-30T06:46:36.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55685bf1-d418-40cf-be15-dc5df7536bd4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144492025-04-30T06:46:36.660ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
144502025-04-30T06:46:36.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6073f13-7f20-46c5-9e50-f429a7c5f479", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144512025-04-30T06:46:36.661ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
144522025-04-30T06:46:36.662ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4334db75-1053-4ddf-86a2-4b5d51420d2b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144532025-04-30T06:46:36.662ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
144542025-04-30T06:46:36.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfd87763-448f-4d05-ab6d-17041585c1e0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144552025-04-30T06:46:36.663ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
144562025-04-30T06:46:36.665ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98d94f97-f2d3-4fee-b9d2-b30193641113", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144572025-04-30T06:46:36.665ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
144582025-04-30T06:46:36.670ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a4f9c62-5f2a-4241-a28b-2f9669b5665e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144592025-04-30T06:46:36.670ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
144602025-04-30T06:46:36.676ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
144612025-04-30T06:46:36.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fed57d8-1dd6-4209-a6fa-dc3584491e07", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144622025-04-30T06:46:36.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
144632025-04-30T06:46:36.683ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9799893b-db6d-4a41-8f70-ece56eb63da5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144642025-04-30T06:46:36.684ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
144652025-04-30T06:46:36.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f09b03f-4459-4df8-bf6f-8ff8038d374f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144662025-04-30T06:46:36.688ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
144672025-04-30T06:46:36.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8c2955c-2a28-4023-a695-fcac405339e0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144682025-04-30T06:46:36.692ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
144692025-04-30T06:46:36.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77cc36d8-d821-475a-954c-37b553dbbe13", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144702025-04-30T06:46:36.694ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
144712025-04-30T06:46:36.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dac1a3c-1093-42d6-bce6-5cfa1fceeb32", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144722025-04-30T06:46:36.696ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
144732025-04-30T06:46:36.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66934b81-bb77-4c12-be83-25baa437d275", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144742025-04-30T06:46:36.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
144752025-04-30T06:46:36.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17de1b7e-04f6-4f7b-a7fe-15ad04dbbfba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144762025-04-30T06:46:36.701ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
144772025-04-30T06:46:36.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f5e2d62-b749-486d-b5e9-c0af2a09a88d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144782025-04-30T06:46:36.703ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
144792025-04-30T06:46:36.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "251c2411-714a-428a-b9f1-952847639a17", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144802025-04-30T06:46:36.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
144812025-04-30T06:46:36.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7084e873-4ba0-44d5-a9b8-7279fca9d3b7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144822025-04-30T06:46:36.709ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
144832025-04-30T06:46:36.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27332be3-5fe5-44f7-953f-193e3653fd64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144842025-04-30T06:46:36.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
144852025-04-30T06:46:36.711ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e816a330-d1d7-4fc5-b5fa-dcba7ac00cb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144862025-04-30T06:46:36.712ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
144872025-04-30T06:46:36.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16274c06-2f90-45f7-a5b0-9b41113757d2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144882025-04-30T06:46:36.714ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
144892025-04-30T06:46:36.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d356c1e2-c608-47ff-9880-3eb9806ddb71", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144902025-04-30T06:46:36.716ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
144912025-04-30T06:46:36.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c69f5178-11a3-4683-bf50-81dff14681f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144922025-04-30T06:46:36.717ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
144932025-04-30T06:46:36.718ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aafea73e-a491-4c35-a28d-29549ad3a40c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144942025-04-30T06:46:36.719ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
144952025-04-30T06:46:36.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "850b1e5c-1ddd-4119-b078-c00f214c64bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144962025-04-30T06:46:36.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
144972025-04-30T06:46:36.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "670912a9-f724-4801-81d2-820487088a7c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
144982025-04-30T06:46:36.722ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
144992025-04-30T06:46:36.727ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145002025-04-30T06:46:36.728ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145012025-04-30T06:46:36.728ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145022025-04-30T06:46:36.728ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145032025-04-30T06:46:36.729ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145042025-04-30T06:46:36.730ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32d25ff2-2d1b-490d-8547-1228aca0422e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145052025-04-30T06:46:36.730ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
145062025-04-30T06:46:36.741ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145072025-04-30T06:46:36.751ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
145082025-04-30T06:46:36.751ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145092025-04-30T06:46:36.751ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145102025-04-30T06:46:36.752ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145112025-04-30T06:46:36.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95786adc-336b-4ca7-947e-caf99a7217e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145122025-04-30T06:46:36.754ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
145132025-04-30T06:46:36.755ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145142025-04-30T06:46:36.755ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145152025-04-30T06:46:36.755ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145162025-04-30T06:46:36.762ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aacdf8b-06d7-416e-b003-e292cf5f2b69", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145172025-04-30T06:46:36.763ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
145182025-04-30T06:46:36.770ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f015c10-ed13-4d41-9950-b37131750910", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145192025-04-30T06:46:36.771ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
145202025-04-30T06:46:36.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3777caa7-ec1d-461e-9b1f-d2f9c2eb347b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145212025-04-30T06:46:36.775ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
145222025-04-30T06:46:36.782ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e3537b-a3dd-437d-bdb6-b587c854663d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145232025-04-30T06:46:36.783ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
145242025-04-30T06:46:36.791ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
145252025-04-30T06:46:36.793ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4dcebec-95fa-47e1-87c0-01b10b04d2af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145262025-04-30T06:46:36.793ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
145272025-04-30T06:46:36.795ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "690dd838-ad45-4d76-a06e-3e123fe7922a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145282025-04-30T06:46:36.795ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
145292025-04-30T06:46:36.796ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ddf49ab-6880-4459-993d-2d9d3168eda7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145302025-04-30T06:46:36.796ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
145312025-04-30T06:46:36.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0bc4e10-15ea-4b99-b86b-b491d54a1eef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145322025-04-30T06:46:36.798ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
145332025-04-30T06:46:36.801ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a9871fe-7d58-4bbe-85e0-48561ff2c7bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:35 GMT"} })
145342025-04-30T06:46:36.801ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145352025-04-30T06:46:36.801ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145362025-04-30T06:46:36.801ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145372025-04-30T06:46:36.802ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145382025-04-30T06:46:36.802ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145392025-04-30T06:46:36.806ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145402025-04-30T06:46:36.813ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145412025-04-30T06:46:36.813ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145422025-04-30T06:46:36.813ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145432025-04-30T06:46:36.813ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145442025-04-30T06:46:36.813ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145452025-04-30T06:46:36.814ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145462025-04-30T06:46:36.818ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145472025-04-30T06:46:36.826ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145482025-04-30T06:46:36.827ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145492025-04-30T06:46:36.827ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145502025-04-30T06:46:36.827ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145512025-04-30T06:46:36.893ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
145522025-04-30T06:46:36.893ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
145532025-04-30T06:46:36.894ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145542025-04-30T06:46:36.894ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145552025-04-30T06:46:36.894ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145562025-04-30T06:46:36.894ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145572025-04-30T06:46:36.895ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145582025-04-30T06:46:36.895ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:62273 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145592025-04-30T06:46:36.897ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:62273 req_id = a37b6326-a825-4881-8af6-79a213436956 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145602025-04-30T06:46:36.897ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 468 local_addr = [::1]:54590 method = GET remote_addr = [::1]:62273 req_id = a37b6326-a825-4881-8af6-79a213436956 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145612025-04-30T06:46:36.897ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37b6326-a825-4881-8af6-79a213436956", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
145622025-04-30T06:46:36.899ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145632025-04-30T06:46:36.899ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145642025-04-30T06:46:36.899ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
145652025-04-30T06:46:36.900ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
145662025-04-30T06:46:36.900ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145672025-04-30T06:46:36.900ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145682025-04-30T06:46:36.900ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145692025-04-30T06:46:36.901ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145702025-04-30T06:46:36.901ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145712025-04-30T06:46:36.908ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
145722025-04-30T06:46:36.908ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145732025-04-30T06:46:36.908ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145742025-04-30T06:46:36.908ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145752025-04-30T06:46:36.911ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145762025-04-30T06:46:36.918ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
145772025-04-30T06:46:36.918ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145782025-04-30T06:46:36.918ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145792025-04-30T06:46:36.919ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145802025-04-30T06:46:36.920ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145812025-04-30T06:46:36.920ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
145822025-04-30T06:46:36.921ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145832025-04-30T06:46:36.922ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:59344 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145842025-04-30T06:46:36.922ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:64490 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145852025-04-30T06:46:36.923ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:59344 req_id = ae4f1b8d-0db4-4184-a662-ea24e52acd64 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145862025-04-30T06:46:36.923ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:64490 req_id = a1fc7a14-a23c-4ad6-b866-8a5619ac04b0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145872025-04-30T06:46:36.924ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1183 local_addr = [::1]:64080 method = GET remote_addr = [::1]:59344 req_id = ae4f1b8d-0db4-4184-a662-ea24e52acd64 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145882025-04-30T06:46:36.924ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1133 local_addr = [::1]:64080 method = GET remote_addr = [::1]:64490 req_id = a1fc7a14-a23c-4ad6-b866-8a5619ac04b0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145892025-04-30T06:46:36.928ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:53394 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145902025-04-30T06:46:36.930ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:33525 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145912025-04-30T06:46:36.930ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:44224 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145922025-04-30T06:46:36.931ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:60678 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145932025-04-30T06:46:36.931ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:53394 req_id = 3ca277ad-d61c-4eb1-8462-e34e0facd87b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145942025-04-30T06:46:36.932ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 582 local_addr = [::1]:64080 method = GET remote_addr = [::1]:53394 req_id = 3ca277ad-d61c-4eb1-8462-e34e0facd87b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145952025-04-30T06:46:36.933ZINFOtest_multiple_snapshots (dropshot): accepted connection kind = pantry local_addr = [::1]:64080 remote_addr = [::1]:46631 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145962025-04-30T06:46:36.933ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:33525 req_id = c94e0b59-ca62-41e0-b92b-d6a273547786 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145972025-04-30T06:46:36.933ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:44224 req_id = 5b5e8408-ce40-4c07-9c9f-8d751e952c54 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145982025-04-30T06:46:36.933ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:60678 req_id = 521fb385-6615-460e-8c1b-5de914eae5f0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
145992025-04-30T06:46:36.934ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 972 local_addr = [::1]:64080 method = GET remote_addr = [::1]:33525 req_id = c94e0b59-ca62-41e0-b92b-d6a273547786 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
146002025-04-30T06:46:36.935ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1005 local_addr = [::1]:64080 method = GET remote_addr = [::1]:44224 req_id = 5b5e8408-ce40-4c07-9c9f-8d751e952c54 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
146012025-04-30T06:46:36.935ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 1115 local_addr = [::1]:64080 method = GET remote_addr = [::1]:60678 req_id = 521fb385-6615-460e-8c1b-5de914eae5f0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
146022025-04-30T06:46:36.935ZTRACtest_multiple_snapshots (dropshot): incoming request kind = pantry local_addr = [::1]:64080 method = GET remote_addr = [::1]:46631 req_id = b1602b49-f352-47c6-bc5e-5a5429ada596 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
146032025-04-30T06:46:36.936ZINFOtest_multiple_snapshots (dropshot): request completed kind = pantry latency_us = 681 local_addr = [::1]:64080 method = GET remote_addr = [::1]:46631 req_id = b1602b49-f352-47c6-bc5e-5a5429ada596 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
146042025-04-30T06:46:36.960ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
146052025-04-30T06:46:36.961ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
146062025-04-30T06:46:36.962ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 442.26799ms iteration = 55 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
146072025-04-30T06:46:36.962ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 56 reason = Signaled
146082025-04-30T06:46:36.962ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
146092025-04-30T06:46:36.971ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146102025-04-30T06:46:36.971ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146112025-04-30T06:46:36.971ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146122025-04-30T06:46:36.972ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146132025-04-30T06:46:36.978ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
146142025-04-30T06:46:36.979ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146152025-04-30T06:46:36.984ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
146162025-04-30T06:46:36.984ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
146172025-04-30T06:46:36.984ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
146182025-04-30T06:46:36.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
146192025-04-30T06:46:36.986ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
146202025-04-30T06:46:36.986ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
146212025-04-30T06:46:36.986ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
146222025-04-30T06:46:36.987ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
146232025-04-30T06:46:36.992ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
146242025-04-30T06:46:36.992ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
146252025-04-30T06:46:36.994ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
146262025-04-30T06:46:36.994ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
146272025-04-30T06:46:36.995ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3b 54 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ;T..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57802 req_id = 06102d76-5a37-42fc-9d61-a11d4879483f
146282025-04-30T06:46:36.995ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15188,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57802 req_id = 06102d76-5a37-42fc-9d61-a11d4879483f
146292025-04-30T06:46:36.995ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
146302025-04-30T06:46:36.995ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146312025-04-30T06:46:36.996ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57802 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 06102d76-5a37-42fc-9d61-a11d4879483f
146322025-04-30T06:46:36.997ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4461, tv_nsec: 796183456 } })
146332025-04-30T06:46:36.997ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
146342025-04-30T06:46:36.999ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
146352025-04-30T06:46:36.999ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
146362025-04-30T06:46:37.000ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7f d5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57173 req_id = bd6c67e6-760e-4527-b81f-864dcc6c1270
146372025-04-30T06:46:37.001ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32725,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57173 req_id = bd6c67e6-760e-4527-b81f-864dcc6c1270
146382025-04-30T06:46:37.001ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
146392025-04-30T06:46:37.001ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57173 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bd6c67e6-760e-4527-b81f-864dcc6c1270
146402025-04-30T06:46:37.064ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
146412025-04-30T06:46:37.065ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
146422025-04-30T06:46:37.065ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146432025-04-30T06:46:37.066ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146442025-04-30T06:46:37.066ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
146452025-04-30T06:46:37.067ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146462025-04-30T06:46:37.067ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 15 1a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55353 req_id = a84119d2-9edc-4055-be50-ee4f40284999
146472025-04-30T06:46:37.067ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55353 req_id = a84119d2-9edc-4055-be50-ee4f40284999
146482025-04-30T06:46:37.068ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
146492025-04-30T06:46:37.068ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5402, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55353 req_id = a84119d2-9edc-4055-be50-ee4f40284999
146502025-04-30T06:46:37.069ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
146512025-04-30T06:46:37.069ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
146522025-04-30T06:46:37.069ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
146532025-04-30T06:46:37.069ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146542025-04-30T06:46:37.075ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146552025-04-30T06:46:37.076ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146562025-04-30T06:46:37.077ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
146572025-04-30T06:46:37.077ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146582025-04-30T06:46:37.082ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "459db604-e99a-4a83-915c-bfe94eb4776e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146592025-04-30T06:46:37.083ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
146602025-04-30T06:46:37.084ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146612025-04-30T06:46:37.084ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
146622025-04-30T06:46:37.085ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146632025-04-30T06:46:37.086ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37f3498-185a-43c5-855d-3324b9accc3a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146642025-04-30T06:46:37.086ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
146652025-04-30T06:46:37.086ZWARNtest_multiple_snapshots (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
146662025-04-30T06:46:37.091ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29cb414c-9208-4de7-8ccf-3ae80dd6b82f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146672025-04-30T06:46:37.092ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
146682025-04-30T06:46:37.102ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "039d03c2-06d2-460c-951c-4b622b25223d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146692025-04-30T06:46:37.102ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
146702025-04-30T06:46:37.104ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "274c3e9b-837e-4196-b889-a6ea2f3b6b03", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146712025-04-30T06:46:37.104ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
146722025-04-30T06:46:37.107ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e16cc2fc-4c63-46cb-bd30-f51a566217d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146732025-04-30T06:46:37.107ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
146742025-04-30T06:46:37.110ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "adc1c3ed-11b3-4988-aa39-97669b0298fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146752025-04-30T06:46:37.110ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
146762025-04-30T06:46:37.112ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26b9f652-977a-4ba0-9f8d-ebd8226b9fe5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146772025-04-30T06:46:37.112ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
146782025-04-30T06:46:37.114ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef0f2d1-86bb-4f9d-bcc0-26f585222ae6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146792025-04-30T06:46:37.114ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
146802025-04-30T06:46:37.137ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b49b4eb0-dc49-4416-b41d-37713eb40df1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146812025-04-30T06:46:37.137ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
146822025-04-30T06:46:37.139ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a7f5200-946a-431d-b0bd-f9b1811fa37a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146832025-04-30T06:46:37.139ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
146842025-04-30T06:46:37.140ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb9dcc00-0173-43b6-9bdd-fd1fe683b892", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146852025-04-30T06:46:37.141ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
146862025-04-30T06:46:37.145ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df40fc9a-008c-43a8-a001-a4991444c50a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146872025-04-30T06:46:37.145ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
146882025-04-30T06:46:37.145ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
146892025-04-30T06:46:37.149ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a39ba254-5e34-4a33-837b-4bee9c1f5b27", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146902025-04-30T06:46:37.149ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
146912025-04-30T06:46:37.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87afe0b0-0492-49f0-9370-388927703a94", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146922025-04-30T06:46:37.154ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
146932025-04-30T06:46:37.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48b77dab-c50c-4e95-80c5-2c8ac45202ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146942025-04-30T06:46:37.162ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
146952025-04-30T06:46:37.165ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57cb9d6d-0c9e-4993-816c-731dbdb32730", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146962025-04-30T06:46:37.166ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
146972025-04-30T06:46:37.169ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f55e175-9e86-45c4-9a49-343b6f6fa107", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
146982025-04-30T06:46:37.170ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
146992025-04-30T06:46:37.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca18122-508d-481a-8c07-9813d2e813cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147002025-04-30T06:46:37.172ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
147012025-04-30T06:46:37.174ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "598b2a20-ee90-4448-b8e1-e3fe20c30551", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147022025-04-30T06:46:37.174ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
147032025-04-30T06:46:37.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "134144a3-4895-47bf-9fd5-2871386c5b76", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147042025-04-30T06:46:37.176ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
147052025-04-30T06:46:37.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ca7530-5751-49bd-a404-7c0a70af7fd8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147062025-04-30T06:46:37.177ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
147072025-04-30T06:46:37.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4eada3f-2958-45bc-993f-6a43e7c2fe6c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147082025-04-30T06:46:37.178ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
147092025-04-30T06:46:37.179ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "431e0dc8-4ae8-4992-a301-32cb6e984f3b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147102025-04-30T06:46:37.180ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
147112025-04-30T06:46:37.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd533850-6a4b-4ae4-9aab-cd4d63d2359a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147122025-04-30T06:46:37.181ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
147132025-04-30T06:46:37.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb510b47-9765-4358-99fa-274efe0f024a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147142025-04-30T06:46:37.183ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
147152025-04-30T06:46:37.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ab10f9-23e8-488c-bbbc-d3077fd0198b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147162025-04-30T06:46:37.185ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
147172025-04-30T06:46:37.188ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71441c81-2832-42d7-b949-51bf877ddccb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147182025-04-30T06:46:37.189ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
147192025-04-30T06:46:37.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce5dffb6-6573-4701-a19f-3731dbe5553e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147202025-04-30T06:46:37.191ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
147212025-04-30T06:46:37.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87725436-40c4-4d70-b870-3a11c54768dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147222025-04-30T06:46:37.193ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
147232025-04-30T06:46:37.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c072063-bc5e-45f8-9047-cbc233c018ff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147242025-04-30T06:46:37.194ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
147252025-04-30T06:46:37.195ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
147262025-04-30T06:46:37.195ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147272025-04-30T06:46:37.195ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147282025-04-30T06:46:37.195ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
147292025-04-30T06:46:37.196ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147302025-04-30T06:46:37.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4f00ee-dc84-40ac-9947-c4d50add094e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147312025-04-30T06:46:37.197ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
147322025-04-30T06:46:37.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aa73377-220d-4145-8183-bdbe33e5f0ba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147332025-04-30T06:46:37.198ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
147342025-04-30T06:46:37.199ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147352025-04-30T06:46:37.204ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
147362025-04-30T06:46:37.204ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147372025-04-30T06:46:37.204ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
147382025-04-30T06:46:37.205ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147392025-04-30T06:46:37.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a64dff-1bcb-4d30-9e87-a40ed90ad6bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147402025-04-30T06:46:37.206ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
147412025-04-30T06:46:37.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba13b5c-aaf6-4ed9-a43e-f5542ff163bb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147422025-04-30T06:46:37.208ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
147432025-04-30T06:46:37.208ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147442025-04-30T06:46:37.208ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
147452025-04-30T06:46:37.209ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147462025-04-30T06:46:37.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "933b4852-a7e0-4af2-9b1f-6d94e758a554", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147472025-04-30T06:46:37.211ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
147482025-04-30T06:46:37.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a97caacb-3862-4966-8a54-8c55358265b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147492025-04-30T06:46:37.214ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
147502025-04-30T06:46:37.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99331593-0ab8-4eae-8040-9f1b39feebe0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147512025-04-30T06:46:37.219ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
147522025-04-30T06:46:37.223ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "109299c1-fd07-4af2-9476-21e781a35d89", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147532025-04-30T06:46:37.223ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
147542025-04-30T06:46:37.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0768c238-c4ff-4008-b862-d09280be504e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147552025-04-30T06:46:37.234ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
147562025-04-30T06:46:37.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7077731-a3f0-46e4-991d-9cc6fd6dd1ee", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147572025-04-30T06:46:37.236ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
147582025-04-30T06:46:37.237ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81900fc0-215c-4170-8edd-f1b3d1f0c479", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147592025-04-30T06:46:37.238ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
147602025-04-30T06:46:37.239ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9e7b724-dda9-48cb-806c-79804e971bbd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147612025-04-30T06:46:37.239ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
147622025-04-30T06:46:37.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a775da9a-ebd1-4d64-961c-d11e4eda7437", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147632025-04-30T06:46:37.242ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
147642025-04-30T06:46:37.243ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
147652025-04-30T06:46:37.245ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "125087da-fb7c-44d5-9cbe-9861d9f53871", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147662025-04-30T06:46:37.245ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147672025-04-30T06:46:37.245ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147682025-04-30T06:46:37.245ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147692025-04-30T06:46:37.245ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147702025-04-30T06:46:37.246ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147712025-04-30T06:46:37.248ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147722025-04-30T06:46:37.254ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
147732025-04-30T06:46:37.254ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147742025-04-30T06:46:37.254ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147752025-04-30T06:46:37.254ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147762025-04-30T06:46:37.254ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147772025-04-30T06:46:37.255ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147782025-04-30T06:46:37.258ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147792025-04-30T06:46:37.262ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
147802025-04-30T06:46:37.262ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147812025-04-30T06:46:37.262ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147822025-04-30T06:46:37.262ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147832025-04-30T06:46:37.315ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
147842025-04-30T06:46:37.315ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
147852025-04-30T06:46:37.316ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
147862025-04-30T06:46:37.316ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147872025-04-30T06:46:37.316ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147882025-04-30T06:46:37.316ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
147892025-04-30T06:46:37.316ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147902025-04-30T06:46:37.317ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:35032 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
147912025-04-30T06:46:37.318ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:35032 req_id = 426d763a-e0eb-482e-8a17-6ee3c0397472 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
147922025-04-30T06:46:37.318ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 737 local_addr = [::1]:54590 method = GET remote_addr = [::1]:35032 req_id = 426d763a-e0eb-482e-8a17-6ee3c0397472 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
147932025-04-30T06:46:37.319ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "426d763a-e0eb-482e-8a17-6ee3c0397472", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
147942025-04-30T06:46:37.323ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
147952025-04-30T06:46:37.323ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
147962025-04-30T06:46:37.323ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
147972025-04-30T06:46:37.325ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
147982025-04-30T06:46:37.325ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147992025-04-30T06:46:37.325ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148002025-04-30T06:46:37.325ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148012025-04-30T06:46:37.326ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148022025-04-30T06:46:37.326ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148032025-04-30T06:46:37.336ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148042025-04-30T06:46:37.336ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148052025-04-30T06:46:37.336ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
148062025-04-30T06:46:37.336ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148072025-04-30T06:46:37.338ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148082025-04-30T06:46:37.348ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148092025-04-30T06:46:37.349ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148102025-04-30T06:46:37.349ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148112025-04-30T06:46:37.352ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148122025-04-30T06:46:37.353ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148132025-04-30T06:46:37.353ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
148142025-04-30T06:46:37.354ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148152025-04-30T06:46:37.389ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
148162025-04-30T06:46:37.390ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
148172025-04-30T06:46:37.390ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 428.804515ms iteration = 56 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
148182025-04-30T06:46:37.391ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 57 reason = Signaled
148192025-04-30T06:46:37.391ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148202025-04-30T06:46:37.391ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148212025-04-30T06:46:37.391ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148222025-04-30T06:46:37.391ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148232025-04-30T06:46:37.392ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148242025-04-30T06:46:37.395ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148252025-04-30T06:46:37.404ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148262025-04-30T06:46:37.404ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148272025-04-30T06:46:37.404ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148282025-04-30T06:46:37.405ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148292025-04-30T06:46:37.409ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
148302025-04-30T06:46:37.409ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
148312025-04-30T06:46:37.413ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
148322025-04-30T06:46:37.420ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a1 c7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65468 req_id = f6219e27-4ade-4a60-91df-2c96f46ec913
148332025-04-30T06:46:37.420ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41415,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65468 req_id = f6219e27-4ade-4a60-91df-2c96f46ec913
148342025-04-30T06:46:37.424ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
148352025-04-30T06:46:37.424ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148362025-04-30T06:46:37.424ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65468 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f6219e27-4ade-4a60-91df-2c96f46ec913
148372025-04-30T06:46:37.425ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4462, tv_nsec: 224331373 } })
148382025-04-30T06:46:37.425ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
148392025-04-30T06:46:37.440ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f8 20 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e . ..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59838 req_id = ea409e9d-92b3-47da-80c8-bba59e827755
148402025-04-30T06:46:37.441ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63520,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59838 req_id = ea409e9d-92b3-47da-80c8-bba59e827755
148412025-04-30T06:46:37.441ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148422025-04-30T06:46:37.441ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59838 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ea409e9d-92b3-47da-80c8-bba59e827755
148432025-04-30T06:46:37.508ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
148442025-04-30T06:46:37.509ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
148452025-04-30T06:46:37.509ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148462025-04-30T06:46:37.509ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148472025-04-30T06:46:37.509ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
148482025-04-30T06:46:37.510ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148492025-04-30T06:46:37.515ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c3 b7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53106 req_id = 1006ca73-ef44-49a9-936c-a6bda00f0f6f
148502025-04-30T06:46:37.515ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50103,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53106 req_id = 1006ca73-ef44-49a9-936c-a6bda00f0f6f
148512025-04-30T06:46:37.516ZDEBGtest_multiple_snapshots (store): query key key = _clickhouse-admin-keeper._tcp
148522025-04-30T06:46:37.516ZERROtest_multiple_snapshots (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50103, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53106 req_id = 1006ca73-ef44-49a9-936c-a6bda00f0f6f
148532025-04-30T06:46:37.516ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148542025-04-30T06:46:37.524ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148552025-04-30T06:46:37.524ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148562025-04-30T06:46:37.524ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
148572025-04-30T06:46:37.525ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148582025-04-30T06:46:37.527ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection background_task = inventory_collection
148592025-04-30T06:46:37.527ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:48945
148602025-04-30T06:46:37.528ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/ignition
148612025-04-30T06:46:37.542ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148622025-04-30T06:46:37.542ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
148632025-04-30T06:46:37.543ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148642025-04-30T06:46:37.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37bd4f8-8538-4647-98bf-5618d6625771", "content-length": "835", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148652025-04-30T06:46:37.545ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0
148662025-04-30T06:46:37.548ZINFOtest_multiple_snapshots (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
148672025-04-30T06:46:37.550ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148682025-04-30T06:46:37.551ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 00c754be-02ac-44f0-8bcb-662d6f52b330 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148692025-04-30T06:46:37.551ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 00c754be-02ac-44f0-8bcb-662d6f52b330 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148702025-04-30T06:46:37.551ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 00c754be-02ac-44f0-8bcb-662d6f52b330 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148712025-04-30T06:46:37.552ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 00c754be-02ac-44f0-8bcb-662d6f52b330 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148722025-04-30T06:46:37.554ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e81f8e-8088-49eb-aaae-b8a7b6b70a2b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148732025-04-30T06:46:37.555ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0
148742025-04-30T06:46:37.557ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 6430 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = 00c754be-02ac-44f0-8bcb-662d6f52b330 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
148752025-04-30T06:46:37.558ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "00c754be-02ac-44f0-8bcb-662d6f52b330", "content-length": "405", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148762025-04-30T06:46:37.558ZDEBGtest_multiple_snapshots (oximeter): client request body = None method = GET uri = http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148772025-04-30T06:46:37.559ZTRACtest_multiple_snapshots (dropshot_internal): incoming request local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f28df5c2-debb-4486-bec5-4e4998330954 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148782025-04-30T06:46:37.559ZTRACtest_multiple_snapshots (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f28df5c2-debb-4486-bec5-4e4998330954 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148792025-04-30T06:46:37.559ZDEBGtest_multiple_snapshots (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f28df5c2-debb-4486-bec5-4e4998330954 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148802025-04-30T06:46:37.560ZDEBGtest_multiple_snapshots (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f28df5c2-debb-4486-bec5-4e4998330954 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148812025-04-30T06:46:37.564ZINFOtest_multiple_snapshots (dropshot_internal): request completed latency_us = 5400 local_addr = 127.0.0.1:55015 method = GET remote_addr = 127.0.0.1:49496 req_id = f28df5c2-debb-4486-bec5-4e4998330954 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
148822025-04-30T06:46:37.565ZDEBGtest_multiple_snapshots (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:55015/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f28df5c2-debb-4486-bec5-4e4998330954", "content-length": "29", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148832025-04-30T06:46:37.565ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 3ae6adf1-675a-4c6a-a06e-473e2878df6e
148842025-04-30T06:46:37.565ZTRACtest_multiple_snapshots (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148852025-04-30T06:46:37.565ZINFOtest_multiple_snapshots (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
148862025-04-30T06:46:37.565ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e4d0c67-5ce0-4aac-806b-14a7ef6478b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148872025-04-30T06:46:37.566ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1
148882025-04-30T06:46:37.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfcdabe0-adee-4fd3-b62d-dfe9d18eac25", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148892025-04-30T06:46:37.571ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0
148902025-04-30T06:46:37.574ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7154291-13a9-4b47-82e9-1ac80ff5754d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148912025-04-30T06:46:37.575ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1
148922025-04-30T06:46:37.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdcf17d2-263e-457b-805c-bfc20bfd6b85", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148932025-04-30T06:46:37.581ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0
148942025-04-30T06:46:37.582ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
148952025-04-30T06:46:37.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80215f70-e21b-4745-a3c4-6520c5e8077e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148962025-04-30T06:46:37.584ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1
148972025-04-30T06:46:37.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "080365ed-4476-466d-8139-3453bf4db258", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
148982025-04-30T06:46:37.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cmpa
148992025-04-30T06:46:37.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "133e07f9-709f-47d6-a4ca-829bd3f6d04f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149002025-04-30T06:46:37.587ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
149012025-04-30T06:46:37.588ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e058f28a-a57b-428f-a3f1-89d0828bbb0c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149022025-04-30T06:46:37.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
149032025-04-30T06:46:37.589ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ec2311-e2be-4d36-a41e-3a72057d2ced", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149042025-04-30T06:46:37.590ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/0/component/rot/cfpa
149052025-04-30T06:46:37.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "600bbbda-c415-416a-b5d4-f8a6a5be3e50", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149062025-04-30T06:46:37.591ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1
149072025-04-30T06:46:37.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d52898-d496-4a01-b0d7-c61dc653569f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149082025-04-30T06:46:37.592ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0
149092025-04-30T06:46:37.594ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0176e7b7-6e7c-49ca-b4a2-4d8760c33719", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149102025-04-30T06:46:37.595ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1
149112025-04-30T06:46:37.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "173e6ff3-191c-403c-bcc9-2bf6268d7fe6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149122025-04-30T06:46:37.596ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0
149132025-04-30T06:46:37.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4b02edd-f3ef-40ed-b0d6-558f1c57dbd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149142025-04-30T06:46:37.598ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1
149152025-04-30T06:46:37.600ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee183328-3bfa-4c24-86d0-8423d9b78f52", "content-length": "170", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149162025-04-30T06:46:37.600ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0
149172025-04-30T06:46:37.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a9bced-151d-433b-80e8-8ece6d740c25", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149182025-04-30T06:46:37.602ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1
149192025-04-30T06:46:37.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d450b73b-aa64-44a4-9bdb-843fef08da11", "content-length": "179", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149202025-04-30T06:46:37.604ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cmpa
149212025-04-30T06:46:37.605ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bef5741-07c4-4a11-a6a5-3816e5abcfd5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149222025-04-30T06:46:37.605ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
149232025-04-30T06:46:37.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a9fa1a5-ee57-430e-8ded-4f35beb34926", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149242025-04-30T06:46:37.607ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
149252025-04-30T06:46:37.608ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b905c3b4-4497-4d0f-8737-ed842c7892c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149262025-04-30T06:46:37.608ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/switch/1/component/rot/cfpa
149272025-04-30T06:46:37.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68874dff-7c71-4439-9147-623869f67fbf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149282025-04-30T06:46:37.610ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0
149292025-04-30T06:46:37.611ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5511517-4d41-406c-b638-7d97f7062cc4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149302025-04-30T06:46:37.611ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0
149312025-04-30T06:46:37.614ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "809cf77f-7be6-41a8-a5a7-c1ef6da235e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149322025-04-30T06:46:37.614ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1
149332025-04-30T06:46:37.616ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3c7134-383e-42ff-b05a-042725a79f4a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149342025-04-30T06:46:37.616ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0
149352025-04-30T06:46:37.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bcb8242-76e1-4762-974c-4667b3f992d6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149362025-04-30T06:46:37.618ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1
149372025-04-30T06:46:37.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81dcb2cd-6954-48ee-9f85-0ec92396cf79", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149382025-04-30T06:46:37.621ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0
149392025-04-30T06:46:37.624ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cef09d27-6d7f-4a86-9c58-848f50f34e85", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149402025-04-30T06:46:37.624ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1
149412025-04-30T06:46:37.626ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff92bcd9-4a5d-46d3-837c-353ac94ce18f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149422025-04-30T06:46:37.626ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cmpa
149432025-04-30T06:46:37.627ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "791fbfff-3fa1-4e04-b9dc-9e7eb918b616", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149442025-04-30T06:46:37.627ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
149452025-04-30T06:46:37.628ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23bbf55f-195f-429d-aa82-4699f077e98c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149462025-04-30T06:46:37.629ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
149472025-04-30T06:46:37.629ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "747b3872-430c-4d52-9452-fe70f389a759", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149482025-04-30T06:46:37.630ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/0/component/rot/cfpa
149492025-04-30T06:46:37.630ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f3bd85-ecd6-4ba4-83d2-31db0e5f044e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149502025-04-30T06:46:37.631ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1
149512025-04-30T06:46:37.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca6256d6-c595-4eb0-bd0e-9ebf2b004cc9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149522025-04-30T06:46:37.633ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0
149532025-04-30T06:46:37.634ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
149542025-04-30T06:46:37.634ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
149552025-04-30T06:46:37.634ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149562025-04-30T06:46:37.634ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
149572025-04-30T06:46:37.638ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149582025-04-30T06:46:37.639ZDEBGtest_multiple_snapshots (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149592025-04-30T06:46:37.639ZDEBGtest_multiple_snapshots (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149602025-04-30T06:46:37.650ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
149612025-04-30T06:46:37.658ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
149622025-04-30T06:46:37.658ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149632025-04-30T06:46:37.659ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
149642025-04-30T06:46:37.659ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149652025-04-30T06:46:37.660ZTRACtest_multiple_snapshots (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149662025-04-30T06:46:37.660ZDEBGtest_multiple_snapshots (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149672025-04-30T06:46:37.660ZTRACtest_multiple_snapshots (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149682025-04-30T06:46:37.660ZTRACtest_multiple_snapshots (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
149692025-04-30T06:46:37.663ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
149702025-04-30T06:46:37.663ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
149712025-04-30T06:46:37.665ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
149722025-04-30T06:46:37.667ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2398e095-8724-478f-b770-8cd74bfeb796", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149732025-04-30T06:46:37.668ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1
149742025-04-30T06:46:37.673ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d97c3bcf-5d2d-47f2-94e1-150fd018067b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149752025-04-30T06:46:37.673ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0
149762025-04-30T06:46:37.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b146c741-2f75-497a-a480-c5867486cbd3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149772025-04-30T06:46:37.677ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1
149782025-04-30T06:46:37.680ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa3e5213-f900-4cef-b917-ca2355b54992", "content-length": "169", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149792025-04-30T06:46:37.681ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0
149802025-04-30T06:46:37.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc6415b-fab1-417a-8065-1ad5c9346e55", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149812025-04-30T06:46:37.687ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1
149822025-04-30T06:46:37.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7a1309a-30f8-428a-a8ee-72e30ee5f8d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149832025-04-30T06:46:37.693ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cmpa
149842025-04-30T06:46:37.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7df1ad91-7135-4eea-ba02-93cb87ee56e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149852025-04-30T06:46:37.695ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
149862025-04-30T06:46:37.697ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26bb545e-cf6a-435c-bddd-d7df4e577f99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149872025-04-30T06:46:37.698ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
149882025-04-30T06:46:37.718ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
149892025-04-30T06:46:37.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6087940d-e0b4-4dc0-a925-7f15ecf6669a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149902025-04-30T06:46:37.720ZDEBGtest_multiple_snapshots (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:48945 method = GET uri = http://[::1]:48945/sp/sled/1/component/rot/cfpa
149912025-04-30T06:46:37.724ZDEBGtest_multiple_snapshots (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:48945 result = Ok(Response { url: "http://[::1]:48945/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ed7c4e3-6ed8-4898-96a8-9dfa35a6d40d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
149922025-04-30T06:46:37.725ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
149932025-04-30T06:46:37.725ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
149942025-04-30T06:46:37.725ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
149952025-04-30T06:46:37.725ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
149962025-04-30T06:46:37.727ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
149972025-04-30T06:46:37.734ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
149982025-04-30T06:46:37.757ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
149992025-04-30T06:46:37.757ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
150002025-04-30T06:46:37.757ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150012025-04-30T06:46:37.757ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150022025-04-30T06:46:37.757ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150032025-04-30T06:46:37.758ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150042025-04-30T06:46:37.769ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
150052025-04-30T06:46:37.770ZTRACtest_multiple_snapshots (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150062025-04-30T06:46:37.770ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
150072025-04-30T06:46:37.770ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
150082025-04-30T06:46:37.772ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
150092025-04-30T06:46:37.772ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150102025-04-30T06:46:37.779ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
150112025-04-30T06:46:37.779ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150122025-04-30T06:46:37.779ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150132025-04-30T06:46:37.780ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150142025-04-30T06:46:37.784ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150152025-04-30T06:46:37.793ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
150162025-04-30T06:46:37.793ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
150172025-04-30T06:46:37.793ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
150182025-04-30T06:46:37.794ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
150192025-04-30T06:46:37.871ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:54590
150202025-04-30T06:46:37.871ZDEBGtest_multiple_snapshots (BackgroundTasks): client request SledAgent = http://[::1]:54590 background_task = inventory_collection body = None method = GET uri = http://[::1]:54590/inventory
150212025-04-30T06:46:37.872ZINFOtest_multiple_snapshots (dropshot): accepted connection local_addr = [::1]:54590 remote_addr = [::1]:32791 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
150222025-04-30T06:46:37.872ZTRACtest_multiple_snapshots (dropshot): incoming request local_addr = [::1]:54590 method = GET remote_addr = [::1]:32791 req_id = 87708ce5-847d-444d-9968-825874d7d9e4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
150232025-04-30T06:46:37.873ZINFOtest_multiple_snapshots (dropshot): request completed latency_us = 702 local_addr = [::1]:54590 method = GET remote_addr = [::1]:32791 req_id = 87708ce5-847d-444d-9968-825874d7d9e4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
150242025-04-30T06:46:37.873ZTRACtest_multiple_snapshots (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
150252025-04-30T06:46:37.873ZDEBGtest_multiple_snapshots (InternalApi): roles roles = RoleSet { roles: {} }
150262025-04-30T06:46:37.876ZDEBGtest_multiple_snapshots (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
150272025-04-30T06:46:37.876ZDEBGtest_multiple_snapshots (BackgroundTasks): client response SledAgent = http://[::1]:54590 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:54590/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "87708ce5-847d-444d-9968-825874d7d9e4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:46:36 GMT"} })
150282025-04-30T06:46:37.883ZDEBGtest_multiple_snapshots (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
150292025-04-30T06:46:37.883ZDEBGtest_multiple_snapshots (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
150302025-04-30T06:46:37.883ZDEBGtest_multiple_snapshots (BackgroundTasks): finished collection background_task = inventory_collection
150312025-04-30T06:46:37.885ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
150322025-04-30T06:46:37.885ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150332025-04-30T06:46:37.886ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150342025-04-30T06:46:37.886ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150352025-04-30T06:46:37.886ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150362025-04-30T06:46:37.911ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150372025-04-30T06:46:37.916ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
150382025-04-30T06:46:37.916ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150392025-04-30T06:46:37.917ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150402025-04-30T06:46:37.917ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150412025-04-30T06:46:37.979ZDEBGtest_multiple_snapshots (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
150422025-04-30T06:46:37.980ZDEBGtest_multiple_snapshots (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
150432025-04-30T06:46:37.980ZDEBGtest_multiple_snapshots (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352
150442025-04-30T06:46:37.981ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
150452025-04-30T06:46:37.982ZWARNtest_multiple_snapshots (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
150462025-04-30T06:46:37.983ZWARNtest_multiple_snapshots (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
150472025-04-30T06:46:37.983ZDEBGtest_multiple_snapshots (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 592.477012ms iteration = 57 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
150482025-04-30T06:46:37.983ZDEBGtest_multiple_snapshots (BackgroundTasks): activating background_task = inventory_collection iteration = 58 reason = Signaled
150492025-04-30T06:46:37.983ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
150502025-04-30T06:46:37.983ZTRACtest_multiple_snapshots (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
150512025-04-30T06:46:37.983ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150522025-04-30T06:46:37.984ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150532025-04-30T06:46:37.985ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150542025-04-30T06:46:37.986ZDEBGtest_multiple_snapshots: attempt to fetch latest inventory collection result = Ok("found")
150552025-04-30T06:46:37.990ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
150562025-04-30T06:46:37.994ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
150572025-04-30T06:46:37.995ZTRACtest_multiple_snapshots (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
150582025-04-30T06:46:37.995ZDEBGtest_multiple_snapshots (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
150592025-04-30T06:46:37.996ZDEBGtest_multiple_snapshots (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
150602025-04-30T06:46:37.997ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 2 table_name = fields_string
150612025-04-30T06:46:37.997ZTRACtest_multiple_snapshots (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
150622025-04-30T06:46:38.002ZDEBGtest_multiple_snapshots (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(51e79bd8-2b58-4bd1-af56-8cf2cb6962ea, 0)]
150632025-04-30T06:46:38.002ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
150642025-04-30T06:46:38.013ZDEBGtest_multiple_snapshots (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = df13fd92-fb86-423f-8ce8-7026f1aca352 n_rows = 1 table_name = measurements_i64
150652025-04-30T06:46:38.013ZTRACtest_multiple_snapshots (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
150662025-04-30T06:46:38.014ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a1 33 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .3..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58209 req_id = cf68c8c8-5049-4696-8ed9-12edf6b051a3
150672025-04-30T06:46:38.014ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41267,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58209 req_id = cf68c8c8-5049-4696-8ed9-12edf6b051a3
150682025-04-30T06:46:38.015ZDEBGtest_multiple_snapshots (store): query key key = _mgs._tcp
150692025-04-30T06:46:38.015ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
150702025-04-30T06:46:38.016ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58209 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cf68c8c8-5049-4696-8ed9-12edf6b051a3
150712025-04-30T06:46:38.016ZDEBGtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 48945, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 4462, tv_nsec: 815657482 } })
150722025-04-30T06:46:38.016ZTRACtest_multiple_snapshots (DnsResolver): lookup_service_targets: looking up SRV target
150732025-04-30T06:46:38.017ZTRACtest_multiple_snapshots (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a9 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50982 req_id = 1b59f922-4a49-4cc4-9b37-74ce47ef543e
150742025-04-30T06:46:38.017ZDEBGtest_multiple_snapshots (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43359,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50982 req_id = 1b59f922-4a49-4cc4-9b37-74ce47ef543e
150752025-04-30T06:46:38.017ZDEBGtest_multiple_snapshots (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
150762025-04-30T06:46:38.018ZDEBGtest_multiple_snapshots (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50982 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1b59f922-4a49-4cc4-9b37-74ce47ef543e
150772025-04-30T06:46:38.087ZTRACtest_multiple_snapshots (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal